User Survey: Customers Rate Their BI Platform Functionality, 2012

Key Findings

• Survey respondents indicate that two business intelligence (BI) platform categories: Information delivery and analysis capabilities are the most used and best reviewed components of BI platforms in line with responses from previous years.
• Search-based BI and predictive analytics both show double digit percentage increases in extensive use over 2011 as organizations expand both the type of data and use of new data to enrich the decision-making process.
• Most BI platform integration capabilities show sharp decline in extensive use. The expansion of business user-led data discovery alongside or replacing traditional BI functions tamps down the use of metadata management and BI infrastructure.
• Online analytical processing (OLAP) use also decreased which can most likely be attributed to the broader use of in-memory analysis that does not require pre-summarized data.
• The war-horses of BI — dashboards, ad hoc query and reporting — continue to expand with BI professionals giving higher satisfaction ratings to these functions year over year. Extensive use of interactive visualization also continues to grow albeit more modestly than anticipated.

Recommendations

• While respondent emphasis and satisfaction is clearly on information delivery and analysis capabilities, you should not dismiss the need for strong integration components in your BI architecture. Gartner recommends a balanced view of all BI platform capabilities adjusted to fit the context of the use cases to be supported and the deployment breadth planned.
• The results shown in this document are subjective customer perceptions and, in some cases, BI platforms are given credit for capabilities Gartner does not believe they have. This research should be used as a guide and should not replace a complete and detailed evaluation, request for proposal, and/or proof of concept to decide which vendor or product best fits your requirements.

Each year, Gartner evaluates the BI platforms market with the ultimate goal of publishing the "Magic Quadrant for Business Intelligence Platforms," the latest version of which was published in February 2012. Part of the process to evaluate vendors is a large user survey of vendor-supplied references and other organizations; leaders from IT, business, or hybrid IT-business disclose their experiences with BI and analytics products and indicate how those products contributed to overall success of their business.

The format of the Magic Quadrant research limits the details of the survey data which Gartner can disclose. The purpose of this research note is to give additional insight into how survey respondents evaluated the functionality of each vendor as defined by Gartner (see Table 1). For a more detailed definition of each capability, see Appendix 1.

To be included in this research note, a vendor must have at least 12 completed reference surveys. If a single vendor had more than one product under evaluation, each product must have at least 12 completed surveys. In this report, only IBM Cognos 8 and IBM Cognos 10 are noted separately. All other vendors are included in aggregate as individual products did not garner enough responses to be evaluated separately. Additionally, Endeca participated in the survey process before it was acquired by Oracle in 4Q11 and is noted separately this year. Criteria for inclusion in the Magic Quadrant is included in Note 1.

The 2012 Magic Quadrant customer survey included vendor-provided references as well as survey responses from users from Gartner's BI Summit and respondents from last year's survey. There were 1,364 survey responses including 120 (8.8%) supplied by reference sources other than vendors. Total respondents increased by 11% from the 2011 survey.

Source: Gartner (August 2012)

All participants were asked to rate vendors on many attributes, using a scale between one (poor) and seven (excellent). These ratings are normalized to a ten-point scale which is shown in all figures and tables to follow.

References remain most satisfied with the information delivery (such as, reporting and ad hoc) and analysis capabilities of BI platforms. Reporting continues to be the most extensively used option, followed by ad hoc query, dashboards, and BI development tools (see Figure 1). The 14 BI functional capabilities are rated from highest to lowest (left to right).

This year, big increases in extensive use were seen in search-based BI (+26.5%) and predictive analytics (+18.7%) which is not surprising given the expansion in attention to "big data" concerns and how these expanded sources of data can be used to enrich the analytic experience. These increases were countered by steep declines in extensive use of metadata management (−20.5%), Microsoft Office integration (−20.4%), BI infrastructure (−15.3%) and BI development tools (−12.6%). OLAP also decreased (−5.4%), most likely attributed to the broader use of in-memory analysis that does not require pre-summarized data. As more companies explore data discovery as an extension to or replacement for traditional enterprise BI platform capabilities, emphasis shifts to end-user-driven analysis which de-emphasizes the traditional integration capabilities of enterprise BI platforms. As data discovery vendors add more infrastructure options, and enterprise BI platform vendors beef up data visualization options, we expect some of these numbers to increase. Microsoft Excel remains a ubiquitous part of the business analysis cycle but better data visualization in BI platforms is reducing the need to integrate to Microsoft Office as extensively as in the past.

Mobile BI was added as a new capability for BI platforms in 2012. While satisfaction is relatively low, and extensive use is less than 10%, survey respondents indicated that investment in mobility for BI is a top priority this year. We can surely expect a broadening of extensive use in next year's survey.

Rating is equal to the mean of means score across vendors for each capability. The percentage axis reflects the mean percentage of respondents claiming extensive use across vendors. The percentage change number at the top of each capability score bar represents the percentage change in extensive use of the functionality in 2012 over 2011.
Chart represents customer perception and not Gartner's opinion.
The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence; N/A = not available; OLAP = online analytical processing
*Mobile BI added in 2012
N = 1,364

Source: Gartner (August 2012)

The survey includes 33 vendors and 34 product lines in the functionality ratings in 2012 (see Table 2). A minimum of 12 references were required but, as you will see, reference counts varied widely: From 12 for 1010data to 104 for Information Builders. We have noted this information so that you can see the breadth of reviews for each vendor or product. The composite product rating is the average of each vendor's functionality ratings. Each functionality category is evaluated on a separate figure to follow. Products that specialize in certain BI platform functions (for example, predictive analytics or dashboards) may score well in some areas and not in others.

The total number of respondents to this survey was 1,364. Since some were for vendors/products that did not make the minimum threshold for inclusion in this research note, this table is based on 1,321 respondents.

Source: Gartner (August 2012)

The top two reasons indicated for choosing a particular BI platform provider were functionality (45.5%) and ease of use for end users (42.6%). There is a difference in attitude between IT and business users — users significantly weighted ease of use higher than functionality, with IT the opposite — but both are closely evaluated. Gartner hears this every day in inquiry; self-service BI is the goal for many organizations and ease of use is a strong criterion to make that goal a reality.

When you plot ease of use ratings against functionality ratings by vendor (see Figure 2), you can easily see which vendors/products are rated "above average" on both vectors. The data discovery firms — Tableau, QlikTech, Tibco Spotfire and Advizor — all score well in both categories. Of the megavendors, only IBM's Cognos 10 product earns that distinction.

Chart represents customer perception and not Gartner's opinion.
N = 1,321

Source: Gartner (August 2012)

Integration Capabilities

Three of four integration capabilities — BI infrastructure, metadata management and development tools — showed a decline in extensive use over 2011 (see Figure 1). Only collaboration showed a small uptick of 1.6%. Many companies use these capabilities without explicitly knowing that they are using them.

BI infrastructure (see Figure 3), including common security, metadata, administration and object model, is extensively used 30% of the time on average. Extensive use ranges from a high of 60% to a low of 7.3%. The highest rated vendor in this category is Bitam at 9.33. The average rating for BI infrastructure is 8.16 and the lowest is 7.14.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence
N = 784

Source: Gartner (August 2012)

Metadata management (see Figure 4), including reuse of common dimensions, hierarchies, measures, performance metrics and report layout objects, is extensively used 15.9% of the time on average. Extensive use ranges from a high of 52% to a low of 0%. The highest rated vendor in this category is AltoSoft, at 9.43. The average rating for metadata management is 7.87 and the lowest is 5.71.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 572

Source: Gartner (August 2012)

BI development tools (see Figure 5), including programmatic and visual development environment to create stand-alone and/or integrated BI applications, is extensively used 34.1% of the time on average. Extensive use ranges from a high of 85.7% to a low of 4.4%. The highest rated vendor in this category is AltoSoft, at 9.29. The average rating for Bi development tools is 8.02. and the lowest is 5.51.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence
N = 880

Source: Gartner (August 2012)

Collaboration (see Figure 6), including the ability of BI users to share, discuss and annotate information, content and results is extensively used 12.9% of the time on average. Extensive use ranges from a high of 46.2% to a low of 0%. The highest rated vendor in this category is InetSoft at 10.0. The average rating for collaboration is 8.06 and the lowest is 5.71.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 521

Source: Gartner (August 2012)

Information Delivery Capabilities

Four of the six information delivery capabilities — reporting, dashboards, ad hoc query, and search-base BI — showed low, single digit increases in extensive use over 2011 (see Figure 1). Microsoft Office integration showed a 20.5% decrease. Mobile BI was introduced in 2012. Survey respondents showed the most satisfaction with this set of functions.

Reporting (see Figure 7), including the ability to create formatted and interactive reports with or without parameters in a schedule, highly scalable distribution environment is extensively used 68.3% of the time on average; the highest of any functional capability in a BI platform. Extensive use ranges from a high of 92.9% to a low of 34.2%. The highest rated vendor in this category is 1010data at 9.21. The average rating for reporting is 8.5 and the lowest is 7.38.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 1,252

Source: Gartner (August 2012)

Dashboards (see Figure 8), including the ability to publish formal, Web-based or mobile reports with intuitive interactive displays of information in context of recipient, is extensively used 42.3% of the time on average. Extensive use ranges from a high of 83.3% to a low of 10.8%. The highest rated vendor in this category is Tableau Software at 9.23. The average rating for dashboards is 8.38 and the lowest is 7.33.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 1,078

Source: Gartner (August 2012)

Ad hoc query (see Figure 9), including the ability to ask questions of the data without relying on IT to create a report, is extensively used 48.7% of the time on average. Extensive use ranges from a high of 80.8% to a low of 0%. The highest rated vendor in this category is Alteryx, at 9.43. The average rating for ad hoc query is 8.46 and the lowest is 5.71.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence
N = 1,112

Source: Gartner (2012)

Microsoft Office integration (see Figure 10), including the ability to use Microsoft Office (particularly Excel) as the interface while using the BI platform as the processing tier, is extensively used 18.6% of the time on average. Extensive use ranges from a high of 42.9% to a low of 12.5%. The highest rated vendor in this category is PivotLink at 9.14. The average rating for Microsoft Office integration is 8.00 and the lowest is 6.72.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 713

Source: Gartner (August 2012)

Search-based BI (see Figure 11), including the ability to search structured and unstructured data sources and map them into a classification structure of dimensions and measures, is extensively used 12% of the time on average. Extensive use ranges from a high of 58.3% to a low of 0%. The highest rated vendors in this category are Pentaho and PivotLink at 10.0. The average rating for search-based BI is 8.15 and the lowest is 0.0.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence
N = 416

Source: Gartner (August 2012)

Mobile BI was introduced as a new platform capability in 2012. Its rating (see Figure 12), including the ability to deliver report and dashboard content to mobile devices in a publishing and/or interactive mode, is extensively used only 4.4% of the time on average. Extensive use ranges from a high of 15.2% to a low of 0% (12 vendors). The highest rated vendor product in this category is IBM Cognos 10 at 9.52. The average rating for mobile BI is 7.77 and the lowest is 0.0.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence
N = 289

Source: Gartner (August 2012)

Analysis Capabilities

Three of the four analysis capabilities — interactive visualization, scorecards and predictive analytics — saw growth in extensive use over 2011 (see Figure 1) with predictive analytics increasing by 18.7%. Only OLAP saw a decline (−5.4%), which is most likely attributable to the broader use of in-memory analysis that does not require pre-summarized data. There is significant attention on this set of capabilities as BI leaders look to provide more users with sophisticated and easy-to-use options when making fact-based decisions.

OLAP (see Figure 13), including the ability to interactively "slice and dice" data to navigate across multidimensional drill paths, is extensively used 29.7% of the time on average. Extensive use ranges from a high of 78.6% to a low of 0%. The highest rated vendor in this category is InetSoft, at 10.0. The average rating for OLAP is 8.31 and the lowest is 7.14.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
BI = business intelligence; OLAP = online analytical processing
N = 732

Source: Gartner (August 2012)

Interactive visualization (see Figure 14), including the ability to display numerous aspects of data more efficiently by using interactive pictures and charts instead of rows and columns, is extensively used 31.8% of the time on average. Extensive use ranges from a high of 75.2% to a low of 0%. The highest rated vendor in this category is 1010data at 10.0. The average rating for interactive visualization is 8.37 and the lowest is 6.73.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 874

Source: Gartner (August 2012)

Predictive modeling and data mining (see Figure 15), including the ability to classify categorical variables and to estimate continuous variables using mathematical techniques, is extensively used 12.5% of the time on average. Extensive use ranges from a high of 38.7% to a low of 0%. The highest rated vendor in this category is JackBe at 9.05. The average rating for predictive modeling and data mining is 7.96 and the lowest is 5.71.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 557

Source: Gartner (August 2012)

Scorecards (for rating, see Figure 16), including the ability to display metrics in a dashboard and applying them to a strategy map that aligns key performance indicators (KPIs) with a strategic objective, is extensively used 19% of the time, on average. Extensive use ranges from a high of 68% to a low of 0%. The highest rated vendor in this category is Bitam, at 9.25. Average rating for scorecards is 8.19. Lowest rating is 6.79.

Chart represents customer perception and not Gartner's opinion. The chart may feature vendors that (in Gartner's opinion) do not deliver the functional capability described.
N = 710

Source: Gartner (August 2012)

This online survey was developed and hosted by Gartner to support "Magic Quadrant for Business Intelligence Platforms." More than 3,400 unique companies were invited to participate, including vendor-provided references, participants in Gartner's BI Summit series and respondents from last year's survey.

To ensure the integrity of the survey data each survey response was checked by company respondent email. For survey responses from unidentified email accounts such as Gmail or Yahoo, the respondent was contacted and had to provide Gartner with a company email address, a company role and other contact information (this amounted to fewer than five responses, all of which were vetted and ultimately included. Only completed surveys were included in the results.

Appendix 1

Table 3 shows Gartner's BI platforms definition of capabilities, by category.

Source: Gartner (August 2012)