Business Intelligence at the Crossroads: The Need for Lean, Agile, and Effective End-user Solutions

Joshua Greenbaum, Principal Enterprise Applications Consulting Fall, 2010

EAC

2303 Spaulding Avenue Berkeley CA 94703

tel 510.540.8655 fax 510.540.7354 josh@eaconsult.com www.eaconsult.com

Table of Contents
Introduction: The Disconnect Between Business Intelligence Tools, the IT Department, and the Business User................................................................................... 1 The Promise and The Reality: The Data Warehouse/Business Intelligence Disconnect.............. 3 Conclusion: Fulfilling the Promise of BI for the Masses................................................................ 7

Business Intelligence at the Crossroads

Introduction: The Disconnect Between Business Intelligence Tools, the IT Department, and the Business User
The history of the capabilities defined under the rubrics of business intelligence, business analytics, and data warehousing is marked by extraordinary promise, and, unfortunately, extraordinary failure to live up to that promise. In particular, the promise of extending business intelligence to the widest possible user base BI for the masses, as some call it was more honored in the breach than otherwise. Despite billions of dollars in customer spending, and myriad technologies, products, and methodologies, there is little doubt that the ability to provide functional, timely, and actionable business intelligence to the broadest possible number of users has yet to be achieved for most companies. Meanwhile, extraordinary growth in the quantity of data inside and outside the enterprise has added an additional challenge to the business intelligence consumer, and has brought the dysfunction of historical business intelligence approaches into even starker relief. In 2006, according to IDC, the amount of digital information created, captured, and replicated totaled 161 exabytes, or 161 million terabytes. Four years later, in 2010, according to a recent IBM report, that quantity has increased almost tenfold. This incredible rate of growth has further exacerbated the disconnect between what has been delivered by business intelligence vendors and what business users actually need. The reasons for this broad, historical disconnect are deeply embedded in the approaches taken by vendors and their customers to provide a framework for understanding and acting on the key issues impacting businesses. That framework characterized by the classic big bang data warehouse project in turn set the stage for an industry replete with over-priced, excessively complex, and largely dysfunctional products and solutions. This dysfunction in turn set up an ongoing disconnect between the IT department that was charged with delivering business intelligence solutions, and the business users that needed to use those solutions. This dysfunction became increasingly acute as the business users needs for real-time analysis to match the increasingly dynamic global business environment ran into the IT/business user roadblock.

Copyright EAC 2010

Business Intelligence at the Crossroads

With IT forced into an unwilling technology gatekeeper role, and business users forced to stand in line for the use of scarce IT resources, the ability of businesses to use the data inside and outside their enterprises began to diminish, especially relative to the exponential growth of the data and the accelerating business case for better business intelligence. Making Business Intelligence Easier to Consume and Easier to Deploy Recent advances in the consumability of business intelligence technology have begun to whittle away at the enormous disconnect between the promise of BI for the masses and the ability of vendors to deliver on that promise. These advances, in particular, have helped significantly improve the ability of the IT department and business users to work closely together to define and deliver business intelligence solutions in a timely, and costeffective fashion. Further advances in the market, including the advent of open source based alternatives to traditional business intelligence offerings, have helped define a cost-structure for business intelligence that is an order of magnitude lower than what was possible only a few years ago. The result is nothing less than the dawning of a new era in business intelligence, one that promises to sweep away the barriers that have prevented companies from fulfilling their potential as data-driven, analytics-oriented, intelligent enterprises. These advances can be characterized by a lower-cost open source technology foundation, a more consumable agile approach to building business intelligence solutions, and an ability to leverage scarce IT resources to work with business users to build the solutions these users need. Additionally, this can all be done cost-effectively and in timeframes that allow companies to optimize business opportunities and avoid business disruptions.

Copyright EAC 2010

Business Intelligence at the Crossroads

The Promise and The Reality: The Data Warehouse/Business Intelligence Disconnect
A landmark report by a major analyst firm in 1996 provides significant insight into the origins of the disconnect between the promise of business intelligence and the reality of what was actually delivered. While the report glowingly described the return on investment (ROI) from a number of data warehouse projects, in retrospect, the cost, complexity, and time to value of these projects were out of line with the timetables and cost-constraints that are more typical of todays business intelligence requirements. Three data points from the report illustrate this problem: The average cost of these projects was $2.2 million ($3.1 million today, adjusted for inflation). The average payback period was 2.3 years, with over 30% experiencing a 5+ year payback period. The majority of respondents reported that their data warehouses consumed enormous resources and remained works in progress for extended periods of time.

In addition, this oft-cited study deliberately obscured the fact that fully 20% of respondents reported a negative ROI, due to a combination of high costs, low data warehouse usage, and payback period greater than three years. As high as the average cost of these projects was, the cost factor pales in comparison to the fact that payback is measured in years, not weeks or months. To the business user trying to understand an immediate opportunity or threat, scaling the business intelligence monoliths erected by leading vendors became a daunting challenge. The disaffection of the business user from the business intelligence solutions in use at most companies reached its nadir in the first decade of the 21st century, as business intelligence spending continued to balloon without an equivalent growth in accessibility. Indeed, industry reports from this period typically cited the exponential growth in size of analytical databases as a positive effect with absolutely no mention of any increase in

Copyright EAC 2010

Business Intelligence at the Crossroads

numbers of users able to make use of the data, for the simple reason that reaching a large quantity of business users was not the primary goal of many of these projects. This growing disconnect is best illustrated by the following anecdote, which involves a real business intelligence vendor and a real customer, both of whom understandably dont want their names publicized: The senior business intelligence executive of a major database vendor, impressed that one of its customers had built one of the largest data warehouses in the U.S. using the vendors technology, scheduled a visit with the customers CEO to discuss the project. With great fanfare, the CEO showed the BI executive the reports, analysis, and other outputs that were available from this multi-million dollar, multi-terabyte data warehouse. Suitably impressed, the executive asked the CEO how many regular users the data warehouse had. The answer stunned the vendor executive: There were only two users, the CEO and the database administrator who managed the data warehouse. While no one would refute the need for a CEO to have access to a wide range of analysis to run his or her business, the story is emblematic of the disconnect between the growth of data warehouses and business intelligence solutions, and the concept of BI for the masses. The IT/Business User Disconnect A major point of failure for the BI for the masses" dream occurs at the intersection of the business users requirements and the IT departments ability to meet those changing requirements in a timely fashion. In survey after survey, and interview after interview, the inability of IT and business users to reach an effective modus operandi is considered one of the essential barriers to the effectiveness and widespread use of business intelligence across the enterprise.

Copyright EAC 2010

Business Intelligence at the Crossroads

This is not necessarily the fault of either the IT department or the business user. This disconnect is more an artifact of the complex nature of the traditional business intelligence environment and the complexity of the tools IT has at its disposal to help business users create the analytics they need in a timely fashion. There have been many attempts to resolve this gap in the tools available to IT, most of which have been concentrated on empowering business users to develop their own reports and analyses without direct IT involvement. While there are many merits to this approach, the fact remains that these solutions still require considerable IT involvement in the setup and creation of both the datasets to be analyzed, as well as the reports themselves. Complex functions such as ETL (extract, transfer, and load), for example, still need to be managed by IT regardless of how comprehensive the end-user tools may be. And in many cases, the highest-value analyses required by the business user are too complex to create using pure business user tools, meaning that IT must still step in and take charge. The bottom line is that, while empowering the business user is a worthy goal, disempowering the interaction between the business user and the IT department can be counter-productive to deriving the greatest value from a business intelligence environment. Re-connecting IT and the Business User: The Pentaho Approach With this history of expense, complexity, excessive time to value, and a disconnect between IT and the business user, the world of business intelligence today craves new approaches that can leverage some of the innovations that have occurred in recent years, and place them at the disposal of enterprises in need of a solution to their business intelligence problems. Pentaho Corporation, a commercial open source business intelligence and data integration vendor based in Orlando, FL, is directing a program it calls the Agile BI

Copyright EAC 2010

Business Intelligence at the Crossroads

Initiative to address these problems and help drive the use of business intelligence to as broad a set of business users as possible. The Pentaho Agile BI Initiative is a combination of technology and business components that help bridge the historic disconnect between the promise of business intelligence and the reality of the solution that has until now been delivered to the market place. Anchored by Pentahos BI Suite Enterprise Edition, the Agile BI Initiative includes: Low-cost, open source solution. End-to-end, integrated BI and ETL capabilities. Full enterprise-level support. Flexibility of on-demand and on-premise deployment. Support for mobile devices as a BI platform. Support for iterative IT and business-user report generation process.

The first four attributes help address the cost and complexity problems that have bedeviled business intelligence. The ability to leverage open source development resources and create an integrated technology platform allows Pentaho to provide their solutions at a much lower cost than non-open source vendors, while offering full enterprise-class deployment features and support. In this regard, Pentaho is following the lead of open source successes such as Red Hat Linux and mySQL, which are provided today as enterprise-class solutions and are considered the top operating system and database management system, respectively, for new enterprise deployments. Offering the customer a choice between on-premise and on-demand deployment acknowledges the need for many companies to keep sensitive data and analysis behind their firewall (the on-premise option), while offering both the potential of a lower and more predictable cost structure (the on-demand option). Pentaho also supports customer choice in terms of deployment platform, by supporting mobile devices in addition to standard desktop devices.

Copyright EAC 2010

Business Intelligence at the Crossroads

The iterative business intelligence development process embodied by Pentahos Agile BI Initiative takes the companys low cost, integrated platform and uses it as the basis for a complete revamping of the IT/business user relationship. The Pentaho BI Suite allows IT personnel to walk through the development process with a business end-user, first by defining the data set and prototyping the data model, and then rapidly prototyping the reports needed by the business user. This is an iterative process that allows the business user to quickly develop the reports and analyses needed to support dynamic business requirements without draining IT of its resources or the company of its valuable capital. This process also allows IT and business users to work more closely together to address the changing needs of the end users. Agile BI is thus able to ensure that BI projects can better stay in sync with the business needs they address. (A deeper and more comprehensive discussion on the topic of Agile BI can be found in a white paper authored by the chief technology officer of Pentaho, James Dixon. The white paper can be found at www.pentaho.com/agile_bi_whitepapers.)

Conclusion: Fulfilling the Promise of BI for the Masses

Pentahos approach goes a long way towards changing the dynamic regarding cost, timeliness, and accessibility of business intelligence for business users across the enterprise. And, by improving the ability of business users and IT departments to work hand in hand, Pentaho sets up a win/win situation for both sets of stakeholders, thus holding the promise of undoing decades of internal conflict between these two core groups. Letting IT achieve hero status in the eyes of the business user isnt just good for the IT department, its also good for the enterprise as whole. And, most important of all, allowing business users to create and distribute the reports they need for managing dynamic business opportunities and avoiding potential problems ensures that core issues like competitive advantage, good governance, and customer satisfaction become imbued in the day-to-day functioning of the company. This is what BI for the masses was always

Copyright EAC 2010

Business Intelligence at the Crossroads

meant to be, and its a requirement that Pentahos approach to business intelligence promises to fulfill. This promise is described in the second white paper in this series, Realizing the Agile BI Opportunity. This white paper can be found at www.pentaho.com/agile_bi_whitepapers and describes how Pentaho customers have been able to use the companys BI Suite Enterprise Edition to make good on the opportunity represented by Pentaho and its Agile BI Initiative.

Copyright EAC 2010