This research details business intelligence customer experiences with 33 vendors based on a survey used in our "Magic Quadrant for Business Intelligence Platforms." It compares vendors based on sales, support and product capabilities as well as effective support of business success criteria.
Key Findings
- Standardization rates have dropped by 7%, with a slight majority (52%) currently identifying their business intelligence (BI) platform software provider as the enterprise standard, down from 56% in 2011. Megavendor standardization rates have also declined but still remain between 60% and 70%.
- Many products – those from small independents and those involved in data discovery in particular – have been adopted by business and IT leaders to augment traditional BI capabilities. Dashboard and scorecard capabilities continue to proliferate throughout firms with multiple vendors delivering that functionality.
- Data discovery vendors continue to fare well in customer ratings but some slippage has been seen as compared to 2011; small independent vendors dominate the top spots on many ratings while large independents are clustered around the mid-points.
- Megavendors continue to be judged as below average on many key measures of customer success including ease of use, functionality and overall customer experience.
Recommendations
- Include product quality, support quality, sales relationship, business outcomes and customer outlook for vendor future in any evaluation on which you embark. Many firms fail to ask about the intangibles including what it's like to interact with a vendor at many levels above and beyond evaluating functional product fit. The overall vendor experience ratings add more color and background to your decision process.
- If you choose to standardize on an enterprise BI platform you have many options depending on company size and the number of users to be deployed. Don't always assume that large suppliers are the only option. Always make sure to pick a vendor that best suits your functionality needs, cost profile and integration requirements.
- BI functional use, along with software and support quality, directly influences a company's satisfaction with its software provider. Make sure you understand the complete software vendor picture by talking to references (and Gartner) to obtain a candid, unvarnished view of what it's like to use these software packages on a daily basis.
Each year Gartner evaluates the BI platforms market with the ultimate goal of creating its "Magic Quadrant for Business Intelligence Platforms" (see Note 1) which was last published on 6 February 2012. Part of that process is a large user survey of vendor-supplied references. Organization leaders – including many from IT, business or hybrid IT-business backgrounds – disclose their experiences with vendors' BI and analytics products and explain how those products and their capabilities (see Appendix 1) contributed to their overall business success.
The format of the Magic Quadrant research limits the details of the survey data Gartner can disclose so the purpose of this document is to give additional insight into how survey respondents evaluated the experiences they have with the 33 vendors and 34 products (IBM has two distinct products) taking part. To be included in this research a vendor must contribute at least 12 completed reference surveys. If a single vendor has more than one product under evaluation, each product must have at least 12 completed surveys. In this report, only IBM Cognos 8 and IBM Cognos 10 are noted separately; all other vendors did not garner 12 responses for more than one product, so therefore are included in aggregate.
The 2012 Magic Quadrant customer survey included vendor-provided references and survey responses from Gartner's BI Summit, as well as respondents from last year's survey. There were 1,364 survey responses, 120 (8.8%) of which were from non-vendor-supplied reference lists. The total number of respondents increased 11% from the 2011 survey.
The number of respondents by vendor/product and in alphabetical order is shown in Table 1.
Table 1. Vendor/Product Responses to 2012 Survey
Source: Gartner (August 2012)
Data Insights
There are three areas of analysis that follow:
- Satisfaction with overall vendor experience
- BI standardization trends
- Customer satisfaction with specific aspects of vendor performance
In summary:
- Small independent vendors (categories are summarized in Note 3) dominate the top spots in many ratings. Their customers rated them very highly on many aspects throughout the survey. Software as a service (SaaS) providers also receive generally positive reviews.
- Data discovery vendors also continue to fare well in ratings but, as a group, their scores dipped slightly from 2011.
- Large independent vendors were often clustered around the average scores with open source providers sometimes above average and sometimes below average scores.
- Megavendors which dominate the market in terms of revenue, continue to be judged below the average for all respondents on many measures of customer success including ease of use, functionality and overall customer experience. There are some signs of improvement – IBM Cognos 10's ratings differ significantly from IBM Cognos 8's – but the majority of ratings put the megavendors below the mean scores.
- Standardization rates have dropped by 7% from 2011's survey results. A slight majority (52%) identify their BI platform provider as their enterprise standard, down from 56% from 2011 through 2010. Megavendors, long the bastions of standardization, have seen their BI standard rate drop to between 60% and 70%. Some of the small independent companies have the highest rates of standardization within this study.
- Many smaller vendors with high deployment numbers in large firms have been adopted by business and IT leaders to augment traditional BI capabilities. Dashboard and scorecard capabilities continue to proliferate throughout firms, with multiple vendors delivering that functionality across the enterprise.
Scoring
Respondents rated the functionality and other attributes of their BI platform on a scale of one to seven, where one is "poor" and seven is "outstanding." Those ratings were then normalized to a 10-point scale which is shown in all figures (see Note 2) and analysis to follow. Specific complex calculations were generated to support the analysis (see Notes 3 and 4).
Satisfaction with Overall Vendor Experience
Respondents to the BI Platforms Magic Quadrant survey assessed their satisfaction with vendors and their products in four key areas:
- Overall customer experience
- Success using vendor's products compared against perception of vendor's future
- Product ease of use compared against strength of overall functionality
- Market understanding versus overall BI platform success and benefit
Gartner chose these categories because they portray a well-rounded view of customer attitudes on sales, support and product capabilities alongside how successful the BI initiatives have been using those vendor's products.
Satisfaction with Overall Vendor Experience
In Figure 1, vendors and products are rated on two dimensions:
- Customer sales experience (sales processes and relationships)
- Customer product experience (product quality and support)
In aggregate, survey respondents were more satisfied with the sales experience than they were with product experience, rating it nearly one point higher overall. The average score for sales experience was 8.09 (out of 10), while the product experience rating averaged 7.18. Both ratings are slight improvements over last year.
The upward trajectory of scores from bottom left to top right indicates there is a correlation between sales and product experience &$150; the better the product experience, the higher the sales experience score (and vice versa). While there are outliers, the trend is unmistakable. Megavendors did not fare well on either measure, indicating there is more work to do to address customer concerns in areas that contribute to these ratings and drive overall satisfaction. Data discovery providers all scored above average on the product experience rating. QlikTech, however, slid below the average rating for sales experience this year.
All open source providers improved their sales experience ratings in 2012, but still remain below average on product experience scores. Large independents are clustered around the average with many small independents gaining high scores for both sales and product experience.
Figure 1. Overall Customer Experience
Source: Gartner (August 2012)
Success Using Vendor's Products Compared Against Perception of Vendor's Future
In Figure 2, vendors are rated on three dimensions:
- View of Vendor Future: On the vertical axis, survey respondents were asked to assess their view of their vendor's future &#!50; whether they were more or less positive about a vendor's future prospects within their firm, and whether that attitude had changed since 2011. Responses ranged from one to four, where one = more concerned about the vendor's future, and four = more positive about a vendor's future. The higher the overall rating, the more positive the respondent is.
- Overall BI Success: On the horizontal axis, overall BI success scores represent a composite rating for product capabilities, support, sales experience, product quality and performance. Each category was weighted equally; the higher the composite score, the more positive the overall experience with the vendor.
- Business Benefit Achieved: The color of each dot represents each vendor's average achievement on business benefits scores as ranked by survey respondents. Red dots represent above average scores and blue dots represent below average.
The average score for view of vendor future is slightly less than 3.0 (it was slightly greater than 3.0 in the prior year), as is the overall BI success rating, which is slightly less than 33 versus greater than 33 in 2011. Customer ratings show megavendors falling below average on both ratings. The exception was IBM Cognos 10 which scored above average on view of vendor future and overall BI success (IBM Cognos 8 was below average on both). Customers of the data discovery tools gave their vendors above average ratings for both future and success measures, as well as above average assessments for achieved business benefit. Most small independents continue to be rated as above average. With a few exceptions, the vast majority of clients which scored above average in BI success also have better than average business benefits.
Figure 2. Satisfaction With Vendors, Change in Future Outlook and Average Business Benefit
Source: Gartner (August 2012)
Product Ease of Use Compared Against Strength of Overall Functionality
In Figure 3, vendors are rated on two dimensions:
- Composite ease of use for both users and developers.
- Composite functionality score, which is the average of all ratings for the 14 core BI requirements.
The top two reasons indicated for choosing a BI platform provider were functionality (45.5%) and ease of use for end users (42.6%). Developer ease of use ranked sixth overall with 16.9%. There is a difference in attitude between end users, who significantly weighted ease of use higher than functionality, and IT users who said the opposite – but both are closely evaluated. Gartner hears this every day in inquiry; self-service BI is the goal for many organizations, and ease of use in all areas is a strong criterion to make that goal a reality.
The average composite score for ease of use was 8.14 with the average composite functionality score hitting 8.16. The data discovery firms all score above average in both categories. Of the megavendors, only IBM's Cognos 10 makes that distinction. Others, including IBM Cognos 8, all score below the average on both measures.
Figure 3. Ease of Use Versus Composite Product Rating
Source: Gartner (August 2012)
Market Understanding Versus Overall BI Platform Success and Benefit
In Figure 4, vendors are again rated on three dimensions:
- Market understanding: A composite rating including a view of their vendor's success compared to the previous year, composite ease of use scores (user and developer) and breadth of use (the sum of all BI activities used). See Note 6 for more information on the calculation. The higher the overall rating, the closer the vendor comes to meeting what Gartner determines to be the market requirements.
- Overall BI success: On the horizontal axis, overall BI success scores represent a composite rating for product capabilities, support, sales experience, product quality and performance. Each category was weighted equally; the higher the composite score, the more positive the overall experience with the vendor.
- Business benefit achieved: The color of each dot represents each vendor's average achievement on business benefits scores as rated by survey respondents. Red dots represent above average scores and blue dots represent below average.
While similar to Figure 2, this figure gets at a higher-order measure of meeting customer demands. Gartner created this market understanding rating to identify which vendors are best at addressing high-value customer requirements. Small independent firms including Bitam, Salient, PivotLink, Prognoz and 1010data did very well in this evaluation, exceeding the scores given to data discovery vendors. Oracle and IBM (with Cognos 10) are the only megavendors that placed above the average score for market understanding.
Recommendation: Include product quality, support quality, sales relationship, business outcomes and customer outlook for vendor future in any evaluation on which you embark. Many firms fail to ask about the intangibles including what it's like to interact with a vendor at many levels above and beyond evaluating functional product fit. The overall vendor experience ratings add more color and background to your decision process.
Figure 4. Market Understanding Versus Overall BI Platform Success Score and Business Benefit Score
Source: Gartner (August 2012)
BI Standardization Trends
There are three figures that depict BI standards. They are:
- Standardization levels by vendor
- Standardization levels based on average company size
- Standardization levels based on average number of users deployed
Standardization Levels by Vendor
The slight majority of survey respondents (52%) have an enterprise BI standard but the rate is slipping and is down 7% (from 56%) from 2011. Figure 5 shows the percentage of customers by vendor (IBM is shown in aggregate for this figure) that have chosen their vendor as their enterprise standard. The vendors with the highest standardization rates are small independents: Bitam, Board, Panorama Software and Targit; which, along with SaaS provider MyDials, all have rates above 75%. Megavendors, along with MicroStrategy and SaaS-provider PivotLink all have standardization rates between 60% and 70%. Of the data discover vendors, QlikTech has the highest standardization rate at 45% which is approximately the same as 2011. Tableau and Tibco Spotfire have increased their standardization level significantly in the past year. Customers of each report that about one third have standardized on these products for enterprise BI.
Figure 5. Percentage of Vendor Customers That Consider Its BI Platform to Be Its Enterprise Standard
Source: Gartner (August 2012)
Standardization Levels Based on Average Company Size
In Figure 6, BI standardization rates are shown by vendor, relative to average company size of survey respondents. Companies both small and large standardize on BI platforms but those with 4,000 employees or less tend to standardize on smaller BI platform providers such as Bitam, Birst, Targit and AltoSoft. Larger companies with more than 6,000 employees tend to standardize on megavendors, large independents or data discovery vendors. Across all survey respondents the average company size was 6,109 employees and the average standardization rate was 52%. Some BI products have very low standardization rates (less than 10%) but are used by much larger firms (more than 10,000 employees) but those most likely complement other installed BI technologies.
Figure 6. BI Standard Versus Average Customer Size
Source: Gartner (August 2012)
Standardization Levels Based on Average Number of Users Deployed
In Figure 7, BI standardization rates are shown by vendor and relative to the average number of users deployed. Across all survey respondents the average number of users deployed was 1,175 and the average standardization rate was 52%.
Recommendation: If you choose to standardize on an enterprise BI platform, you have many options, depending on company size as well as number of users to be deployed. Don't always assume that large suppliers are the only option. Customers of nearly every vendor in this survey have standardized on that vendor's platform. Always make sure to pick a vendor that best suits your functionality needs, cost profile and integration requirements.
Figure 7. BI Standard Versus Average Number of Users
Source: Gartner (August 2012)
Customer Satisfaction With Specific Aspects of Vendor Performance
There are two figures that depict other aspects of overall vendor performance. They are:
- BI platform functional usage
- Rating BI platform software quality versus support rating
Whether you're buying, upgrading, standardizing or augmenting BI capabilities you need to know what other customers think of the product and support quality. What looks shiny and slick during a demo may be more difficult to implement, especially if the code driving the software is flawed and/or support can't help you figure out how to fix the issue. Additionally, some products are used more for a specific set of functions, so it's a comfort to know what capabilities clients really use.
BI Platform Usage
BI platforms perform a variety of analytic functions. We asked survey respondents to estimate what percentage of users in their organizations were utilizing eight specific functions (see Table 2).
Figure 8 depicts which functions customers use in each BI platform. SaaS-based vendors 1010data and PivotLink lead the list with the most functions in use across their user base, followed by small independents including Salient, Bitam, Alteryx and Prognoz. Oracle (along with its recent acquisition, Endeca) shows the highest breadth of use of the megavendors. The narrowest usage was reported by Domo with its specialties (dashboards and scorecards) dominating its functional use cases.
Table 2. Eight BI Functions With Average Use Across All Vendors
Source: Gartner (August 2012)
Figure 8. Percentage of Customers Using Vendors for Distinct BI Capabilities
Source: Gartner (August 2012)
Rating BI Platform Software Quality Versus Support Rating
No megavendor was rated at or above average for software quality or support (see Figure 9). Customers of small, independent vendors report the best quality support and software. Data discovery vendors, with the exception of QlikTech, were rated above average. Large independent companies Information Builders and SAS Institute were both rated above average while MicroStrategy slipped below average this year. Open source providers were also judged below average on these important measures. The combination of low software quality and low support measures are a dangerous combination for any vendor.
Recommendation: BI functional use, along with software and support quality, directly influences a company's satisfaction with its software provider. Make sure you understand the complete software vendor picture by talking to references (and Gartner) to obtain a candid, unvarnished view of what it's like to use these software packages on a daily basis.
Figure 9. Rating BI Vendors on Support and Software Quality
Source: Gartner (August 2012)
Methodology
This online survey was developed and hosted by Gartner to support the "Magic Quadrant for Business Intelligence Platforms" analysis. More than 3,400 unique companies were invited to participate including vendor-provided references, participants in Gartner's BI Summit series and respondents from last year's survey.
To ensure the integrity of the survey data, each survey response was checked by company respondent email. For survey responses from non-identified email accounts such as Gmail or Yahoo accounts, the respondent was contacted and had to provide Gartner with a company email address, a company role and other contact information (this amounted to fewer than five responses, all of which were vetted and ultimately included). Only completed surveys were included in the survey results.
Appendix 1
Table 3 shows Gartner's BI platforms definition of capabilities, by category.
Table 3. Platform Definitions by Category
Source: Gartner (August 2012)
Appendix 2
Table 4 shows how Gartner defines each category of BI vendor.
Table 4. Vendor Categories
Source: Gartner (August 2012)
Evidence
The survey was conducted over a four-week period in 4Q11, hosted and executed by Gartner. Summarized results were used as data for the Gartner "Magic Quadrant for Business Intelligence Platforms" published in 2012. This provides details on how survey respondents rate the functionality of 34 products from 33 vendors.
Note 1
2012 BI Platforms Magic Quadrant Inclusion Criteria
To be included in the BI Platforms Magic Quadrant for 2012, the following criteria must be met:
- Vendors must generate at least $15 million in BI-related software license revenue annually. Gartner defines "total software revenue" as revenue that is generated from appliances, new licenses, updates, subscriptions and hosting, technical support and maintenance. Professional services revenue and hardware revenue are not included in total software revenue.
- Those that also supply transactional applications must show that their BI platform is used routinely by organizations that do not use their transactional applications.
- Vendors must deliver at least nine of 14 capabilities detailed in the BI platform capabilities table (see Appendix 1).
- They must be able to obtain and supply a minimum of 30 survey responses from customers that use the vendor's product as an enterprise BI platform.
Note 2
Graphic Figures
The figures in this report include vendors and products with at least 12 survey responses. When individual products of a single vendor garnered more that 12 distinct responses, they are noted separately, except where noted in the text. IBM Cognos 8 and IBM Cognos 10 were the only products to fall in this category. Endeca, acquired by Oracle in 4Q11, is included separately as its customers participated in the reference and survey process prior to the acquisition being announced.
Participants in the survey came from these regions:
- North America – 54%
- Western Europe – 27%
- Rest of the World – 19%
Note 3
Customer Experience Score Calculation
We computed the combined customer support and product quality scores to arrive at a Customer Experience Score as follows:
Vendor support is scored on a scale of one to seven:
- One to two = Poor
- Three to five = Average
- Six to seven = Outstanding
Product quality is scored on the same basis. We converted these scores to a percentage (vendor score divided by seven). We averaged the percentage, as well as the percentage of respondents reporting no software problems, and normalized the result to a scale of 10 to derive the composite score.
Note 4
Market Understanding Calculation
The market understanding score is computed as an average of the following scores for each vendor:
- View of vendor success in organization compared to 12 months ago: One = less successful, two = no change, three = more successful, normalized to 10. More successful is defined in the survey as "BI platform is being used more widely or with greater sophistication." Less successful is defined in the survey as "BI being used by fewer users, or being replaced by other tools."
- Composite ease of use scores, normalized to 10.
- Breadth of use: Sum of user activities (see Table 1 for list of functions), normalized to a base of 10.
Source; Gartner Research Note G00227584, J. Hagerty, 14 August 2012
