Survey Analysis: It's Time to Push Your Data Integration Tools Beyond the Basics

Sound data integration capabilities are vital to unlock the value in information. This report highlights key data integration tool functions, emerging data delivery styles and other parameters, such as pricing, that are critical to customer satisfaction.

Key Findings

• Modern data delivery styles such as data virtualization are gaining momentum and give vendors opportunities to differentiate their offerings. However, more than 50% of organizations are still not investigating these styles and may be missing out on key opportunities, such as to enable a logical data warehouse (LDW) and increase the flexibility of their overall information infrastructure.
• Increasingly, data integration tool vendors offer capabilities such as support for big data integration, support for integrating message-oriented data (including data from events and streams) and application integration features in a single or integrated product set.
• Most vendors still offer only hardware-based licensing options, which are difficult to comprehend and benchmark and can prove highly cost-inefficient as the scale of deployment increases. This is making many customers more dissatisfied with pricing.

Recommendations

Information leaders responsible for data integration should:

• Investigate modern data delivery styles, such as data federation/virtualization, as useful components of an overall data integration technology portfolio, and look for ways to complement or extend their existing data integration architectures.
• Study their incumbent data integration vendor's roadmap to see whether capabilities such as support for big data integration, data quality, application integration and cloud integration will be addressed. If their incumbent vendor has no plans to support these, they should evaluate other vendors' offerings.
• Evaluate data integration tool vendors that offer flexible and easily understood pricing methods, such as subscription-based and named-user plans for new requirements. They should also consider open-source data integration tools. 

Survey Objective

In 2Q15, Gartner surveyed 348 organizations in different regions about their data integration initiatives and deployment of data integration tools. These organizations were reference customers identified by vendors for our study of the adoption and use of data integration tools.

In this document, we use findings from this survey to highlight key adoption trends involving data integration tool functions and both traditional and modern data delivery styles. We also highlight some key areas on which vendors could focus to improve customer satisfaction in terms of development, deployment and ongoing support of their existing data integration tools.

For more information on our methodology and the composition of the respondent base, see the Methodology section.

Data Insights

Data Integration Grows in Importance

Data integration is central to enterprises' information infrastructure and remains one of their biggest areas in terms of both expenditure and concern. Attempts to enable integrated digital business will add further complexity to data integration strategy by requiring a mix of latencies and patterns, as well as deployments using a combination of on-premises and cloud-based models. As a result, more information leaders are realizing that they need to investigate and employ a diverse portfolio of data integration capabilities and data delivery styles in order to share data across all organizational and system boundaries.

This growing interest in data integration is reflected in the inquiries we receive from Gartner clients: we have answered over 6,000 inquiries about data integration in the past two years (see the Evidence section). This data clearly shows the growing importance of data integration to Gartner clients, their concerns about data integration, and its continued relevance to ensuring the robustness of information infrastructure in the era of digital business, the Internet of Things (IoT) and big data.

Overview of the Market for Data Integration Tools

Gartner estimates that the data integration tool market was worth approximately $2.4 billion in constant currency at the end of 2014, an increase of 6.9% from 2013. We estimate that it will grow by an impressive 8.9%, to $2.5 billion, by the end of 2015, which highlights the growing importance of sound data integration practices for successful digital business. For more information on the data integration tool market and an overview of some popular and established data integration tools.

Adoption Trends for Key Data Integration Tool Functions and Data Delivery Styles

We asked the survey respondents whether their enterprise currently uses, or plans to use, a number of data integration functions (for clarifications of styles, functions and capabilities, see Note 1). Most of the functions shown in Figure 1 are aligned with the four major styles of data integration:

• Bulk/batch data delivery
• Data federation/data virtualization
• Message-oriented movement via encapsulation
• Data replication and synchronization

The results show a wide disparity in the use of the various functions of data integration tools.

Figure 1. Use of Key Data Integration Functions and Capabilities
Survey Question: Does your organization currently use, or plan to use, a vendor for any of the following data integration tool functionalities?

Number of respondents = 348
DI = data integration; iPaaS = integration platform as a service
Source: Gartner (December 2015)

Bulk/Batch Data Delivery Remains the Dominant Delivery Style, but Opportunities Exist to Exploit Nontraditional Styles Like Federation/Virtualization, Message-Oriented Movement, and Replication and Synchronization

Unsurprisingly, an overwhelming majority (86%) of respondents said that their enterprise currently uses or plans to use the most common style, bulk/batch data movement (see Figure 1). There were much lower adoption levels for data federation/virtualization and message-oriented movement: 25% and 28%, respectively, said that their enterprise either uses or plans to use these functions in the near future. However, data federation/virtualization has gained significant momentum in terms of usage over the past few years, and it is increasingly used by organizations for production-level use cases (integrating big data analytics and virtual sandboxes, prototyping, and implementing an LDW, for example). Many vendors now offer core virtualization-specific data integration tools (examples are Cisco and Denodo).

These results also indicate that over 50% of organizations that do not plan to use these modern data delivery styles with their data integration tools may be missing opportunities to improve the overall flexibility of their information infrastructure, or to pursue new and possibly transformational use cases. For instance, many companies are using data federation/virtualization capabilities to enable an LDW by providing the required integrated views of data across the various parts of an LDW architecture.

The survey showed more balanced results for data replication and synchronization (63% of the respondents use or plan to use this). The increasing use of replication/change data capture (CDC) could indicate growing interest in integrating real-time information environments in view of digital information strategy, the IoT and the Nexus of Forces. Replication/CDC is also being used extensively by organizations to offload their production data warehouse to other data stores (Hadoop data stores or data lakes, for example) for enhanced performance. Increasingly, data delivery will involve data from other parts of the enterprise and from outside its organizational boundaries, which will require companies to integrate data not only from their traditional data warehouses or databases but also from external sources, in near real-time, for operational agility.

Buyers Can Exploit the Data Quality Capabilities of Their Existing Data Integration Tools to Drive Synergies

For 60% of the respondents, efforts to improve data quality and/or data governance remain relevant as part of data integration work. Over the years, Gartner has seen a growing trend for those buying – and those providing – data integration technologies to seek to combine them with data quality tools in a coherent solution set. In 2015, many purchases of data integration tools included data quality functions from the same vendor. Toolsets offering comprehensive and well-integrated functionality for data integration, data quality and governance have become more mature and robust, and are now readily available.

However, 28% of the respondents said that their enterprise either does not use or does not plan to use their data integration tool's data quality and governance functionalities. This is the case even though doing so would enable these capabilities to interoperate, and help support data quality and governance use cases. Possible reasons for this finding are that some companies already have a separate data quality tool catering to their data quality use cases and therefore do not evaluate or harness their data integration tools' data quality functions. In most cases, companies take a tactical approach to their data quality strategy – in fact, it is often an afterthought as they do not realize its importance for creating high-quality data assets. The onus is now on information leaders to investigate and use the data quality and governance functionalities of their existing data integration tools in order to avoid redundant purchases and exploit synergies.

Opportunities Remain to Exploit Capabilities for Big Data Initiatives

Another interesting finding is that more than one in three respondents (34%) make some use or extensive use of their existing data integration tools and platforms for supporting big-data-related use cases. The 51% of respondents who do not plan to use this functionality may be missing an opportunity here, since integrating big data has become critical (this includes the ability to support the delivery of data to, and access data from, the platforms typically associated with big data initiatives, such as Hadoop and NoSQL data stores).

Data integration tool vendors such as Informatica, Syncsort and Talend now offer mature big data integration functionalities as part of their data integration portfolios. These are enabling companies to integrate with, and use, big data repositories in near real-time.

Gartner urges companies to investigate the big data functions of their data integration tools in order to improve support for their big data initiatives and related use cases.

Synergies Exist Between Data Integration and Application Integration

Over the years, Gartner has highlighted the importance of converging data integration and application integration to avoid counterproductive practices and escalating deployment costs, as these two disciplines are synergistic in nature. The survey found that over one in three respondents (34%) already use their existing data integration tools to interoperate with application integration tools. The 45% who reported otherwise must start investigating and exploring synergies between application integration and data integration in terms of both technologies and practices, where their use cases intersect. They must align technologies with a combination of capabilities, such as enterprise service bus (ESB), data integration technologies and hybrid integration platforms, to achieve shared resources, superior ROI and overall easier management of vendors.

Interest in iPaaS Model Increases as Deployment Approaches Diversify

Adoption of cloud-based integration approaches such as integration platform as a service (iPaaS) continues to grow, with almost 23% of the respondents reporting that they use this capability in some capacity (up from 20% last year). However, on-premises deployments are still the most popular choice, with almost 60% of the respondents reporting that they do not plan to adopt iPaaS or other similar cloud-based integration approaches. This means they will miss out on the potential advantages that cloud delivery models for data integration tools can bring to organizations. These include scalability, elasticity and a significantly reduced entry-level cost of deploying data integration infrastructure.

However, it is interesting to note that some of the newer customers of data integration tools (those whose data integration tools have been in production for less than one year) make extensive use of iPaaS and other cloud-based integration approaches. This indicates that cloud delivery models appeal particularly to organizations with limited resources that need to address common integration tasks, deploy tools rapidly and add functionality easily.

Recommendations to information leaders responsible for data integration:

• View data integration and data quality technology capabilities as synergistic and foundational to your information management infrastructure. Combine related data integration and data quality projects, and support them with a single toolset, if possible. Designing processes jointly will lead to more consistent results.
• Assess whether you are exploiting the potential advantages of converging data integration and application integration. Prioritize effort and investment in the overlapping areas. This should result in shared skills and improved ROI, and make it easier to manage vendors.
• Investigate potential use cases for, and the benefits of, integrating big data repositories (Hadoop and NoSQL repositories, for example). Extend, rather than replace, your current data integration tools and architectures to support big data environments with the same functions that are used in traditional environments, so as to make fuller use of investments you have already made.
• Consider iPaaS as an extension of your data integration infrastructure to manage cloud-related data delivery. Assess how purpose-built iPaaS offerings can solve data integration problems when increased developer productivity, lower initial cost of solution or shorter time to market are key drivers.

Customer Satisfaction With Key Data Integration Functions

Our survey then asked respondents to rate their data integration vendors, based on how well the tools' functions fulfill their enterprise's needs (see Figure 2). Overall, the results show a high degree of satisfaction with data integration tool vendors: the average rating for all functions together was 5.5 (well-satisfied).

The satisfaction ratings relate directly to the maturity of the functions and capabilities of the data integration tools. The mature bulk/batch data delivery and data transformation functions have the highest satisfaction ratings, 6.06 (out of 7) and 6.03, respectively. The lowest satisfaction ratings are for the less mature functions of metadata management and modeling (5.41) and federated and/or virtualized views formulated in memory (5.44); as these functions mature and become more broadly supported in data integration tools, we expect that their satisfaction ratings will rise.

Figure 2. Mean Scores and Detailed Results for Satisfaction With Tool Functions
Survey Question: How well do you think vendor is fulfilling, or expect that it will fulfill, your organization's needs for each of the following data integration tool functionalities?

DI = data integration; iPaaS = integration platform as a service
Source: Gartner (December 2015)

Opportunities Exist for Vendors to Differentiate Their Offerings on Modern Data Delivery Styles

A detailed look at the ratings in Figure 2 indicates that data integration tool vendors have more opportunities to differentiate themselves on key integration functionalities than they realize. Organizations report some degree of dissatisfaction (ratings of 4 or below on a scale of 1 to 7) with some of the more modern styles of data delivery, such as data virtualization and message-oriented movement, and with some of the relatively new features of their data integration tools, such as support for big data initiatives. For example, 17% of the surveyed users of message-oriented movement capabilities rated their satisfaction at 4 or below. Similar proportions expressed some degree of dissatisfaction with federated or virtualized data delivery styles (17%) and delivery of data integration capabilities as cloud services (17%).

For a detailed vendor-focused breakdown of assessment scores across all data delivery styles and other functionalities and capabilities.

Established Vendors Score Well for Traditional Data Delivery Styles but Poorly for Modern Data Delivery Styles and Functionalities

When we looked at the survey results for new users of data integration tools whose projects are still in the development phase, we uncovered some more interesting findings. Most of these users report some degree of dissatisfaction with traditional data integration tools' delivery of capabilities as cloud services (or iPaaS), with an average rating of just 4.8. Two lesser-used forms of data delivery – data federation/virtualization and replication/synchronization – also scored relatively poorly, with ratings of 4.29 and 5.06, respectively. This indicates that although traditional and more established data integration tool vendors do exceedingly well in the more traditional forms of data delivery, such as bulk/batch movement of data, their modern functionalities, such as virtualization of data and cloud delivery of data integration, need further improvement. This is where some of the new vendors are supporting customers better with offerings focused on these types of delivery and functionality.

These findings show that data integration tool vendors have opportunities to improve their product offerings in terms of these data delivery styles and functions, and to achieve some differentiation in this otherwise mature technology market.

Recommendations to information leaders responsible for data integration:

• Re-evaluate your incumbent vendors if you are not highly satisfied with the existing functions they provide or if you need to add new functions. Some incumbent vendors' tools are excellent for traditional ETL, but might lack functions like big data integration support and/or support for application integration or data virtualization.
• Do not restrict yourself to a data integration tool vendor just because it is familiar or a contributor to a larger technology stack, or because it would be painful to switch. Tools with a limited range of functions may work for current use cases and initiatives (ETL and replication, for example), but more challenging and transformative future use cases will require robust tools with more functions (for application integration, cloud data integration and integration with real-time data, for example).

Customer Satisfaction With Development, Deployment and Support of Existing Data Integration Tools

Some of the survey questions probed customers' satisfaction with their data integration tools and vendors during development, deployment, ongoing operations and support. The responses highlight areas beyond the functions themselves, to which information leaders should pay particular attention.

The survey asked respondents to gauge how well their vendor supported their deployment of data integration tools, based on 10 factors (see Figure 3). The vendors scored 5.98 out of 7, on average, for overall satisfaction – a result almost in the zone of loyalty. Results for individual deployment areas also show high degrees of satisfaction.

Figure 3. Mean Scores and Detailed User Satisfaction Ratings for Support for Data Integration Tool Deployments
Survey Question: How satisfied is your organization with your vendor's performance in each of the following areas?

Number of respondents = 348
Source: Gartner (December 2015)

Users Express Significant Dissatisfaction With Vendors' Prices (Actual Cost of Data Integration Tool Relative to Its Value) and Pricing Methods

Vendors' pricing methods and actual prices – the cost of the tools relative to the value they deliver – stand out for having relatively low mean ratings of 5.30 and 5.49 out of 7, respectively (see Figure 3).

Approximately 22% of the surveyed users rated their products 4 or below – which is in the zone of switching or dissatisfaction – for the vendor's actual prices. Last year, the average score for this was 5.50, and its drop to 5.30 clearly indicates decreased satisfaction with the cost of data integration tools relative to their perceived benefits. This is an area of focus for vendors aiming to differentiate themselves and achieve a superior rate of customer retention.

Another important finding is that customers' satisfaction with the pricing methods used for data integration tools has continued to decline. The mean score of 5.30 conceals this growing concern somewhat, but on closer inspection we see that 25% of the surveyed users are dissatisfied with their vendor's pricing method (scores of 4 or less). This is a telling figure and exactly in line with what Gartner gathers from hundreds of inquiries received from, and discussed with, clients. Most vendors still provide only hardware-based licensing options (based on CPUs, cores, processors, servers and processor speeds). Customers find these difficult to comprehend and benchmark; increasingly, they are left frustrated as this form of pricing is not flexible and can be extremely expensive as the scale of deployments increases.

With the growing popularity of open-source integration platforms (from CloverETL and Talend, for example) and iPaaS tools, clients are increasingly looking at term-based and subscription-based licensing options to go along with standard perpetual models. This combination offers greater flexibility and more control.

Vendors' Product Support and Version Upgrades/Migrations Need Further Improvement

Figure 3 also offers a more detailed look at factors relating to the deployment and operation of data integration tools. More potential risks stand out.

Surveyed users reported some dissatisfaction with version upgrades or migrations of their data integration tools, with 19% awarding scores of 1 to 4. Upgrades and migrations are increasingly reported as "pain points" by customers, and the treatment of these could mean the difference between customers switching vendor or staying with their current vendor.

Also, 17% of the respondents expressed some degree of dissatisfaction (scores of 1 to 4) with their vendor's technical support and documentation, which are crucial for overcoming software bugs and undertaking complex deployments.

Between 10% and 20% of the respondents reported dissatisfaction with almost all the other factors.

Recommendations to information leaders responsible for data integration:

• Do not let your overall satisfaction with your current vendor distract you from potential gaps in important aspects such as product roadmap, version migration and upgrades. For a better total cost of ownership, place greater emphasis on vendors' product support and pricing components when evaluating vendors for long-term engagements.
• Investigate vendors that offer open-source data integration products or iPaaS for trying out new use cases or one-off projects. Also, ask your incumbent vendor for its pricing roadmap and inquire about flexible pricing options, such as "named user," "unlimited licensing" and subscription-based.

Survey Demographics

Gartner surveyed 348 data integration professionals in April 2015 as part of our research into the market for data integration tools. More than two-thirds of the respondents came from North America, one-fifth from Western Europe and the rest from other regions (see Figure 4).

Figure 4. Breakdown of Respondents by Region

Number of respondents = 348
Source: Gartner (December 2015)

We surveyed respondents from enterprises that have deployed packaged data integration tools in many different industries (see Figure 5). The respondents also came from enterprises of all sizes (see Figure 6).

Figure 5. Breakdown of Respondents' Enterprises by Industry

Number of respondents = 348
Source: Gartner (December 2015)

Figure 6. Breakdown of Respondents' Enterprises by Number of Employees

Number of respondents = 348
Source: Gartner (December 2015)

The survey respondents represented a range of spending on data integration tools (see Figure 7). More than one-quarter said their enterprise spends less than $100,000 per year on such tools. These could be enterprises just starting out by implementing a simple integration use case for ETL, for example, by deploying a data integration tool using a flexible subscription-based pricing model. Alternatively, these enterprises may have already successfully implemented data integration tools and may now be paying only the standard maintenance or subscription fee.

At the other end of the spectrum, one-tenth of the respondents work for enterprises that reportedly spend more than $1 million on data integration tools each year. This is in line with what Gartner observes to be the standard deal size for large-scale deployments of data integration tools using the conventional (and still popular) perpetual, on-premises delivery model.

Figure 7 shows that data integration is still a major market for customers in terms of expenditure, as they are finding that data integration remains one of the biggest challenges. This challenge will grow in the era of digital business, the IoT and big data.

Figure 7. Breakdown of Respondents' Annual Spending on Data Integration Tools

Number of respondents = 348
Source: Gartner (December 2015)

Methodology

The analysis in this report is based on information from a number of sources, including:

• A Web-based survey of reference customers – identified for Gartner by their vendors – conducted as part of Gartner's 2015 study of the market for data integration tools. The study captured data on, among other things, usage patterns and customers' satisfaction with major product functionality categories and vendors' nontechnology attributes (such as pricing, product support and overall service delivery). In total, 348 organizations across all major regions provided input on their experiences with data integration vendors and tools in this manner.
  • The data in this report represents the views of the survey respondents; it does not represent Gartner's ratings of vendors or their products.
  • To ensure the integrity of the survey data, responses were verified by company email address. For survey responses from unidentified email accounts, each respondent was contacted and had to provide Gartner with a company email address, a company role and other contact information. Only completed surveys were included in the results.
• Feedback about tools and vendors captured during conversations with users of Gartner's client inquiry service.
• Interactive briefings in which the vendors provided Gartner with updates on their strategy, market positioning, recent key developments and product roadmap.
• Market share estimates developed by Gartner's Technology and Service Provider research unit.

Definitions

Data Integration

The discipline of data integration comprises practices, architectural techniques and tools for achieving consistent access to, and delivery of, data across the full spectrum of data subject areas and data structure types within enterprises.

Data integration is necessary to meet the data consumption requirements of all applications and business processes.

Data integration capabilities are at the heart of Gartner's Information Capabilities. They enable the sharing of data across organizational and system boundaries.

Application Integration

Application integration is the process of giving independently designed applications the ability to work together.

Evidence

Data Integration Remains a Key Area of Interest for Gartner Clients

In 2014 and 2015 (so far), the number of inquiries about data integration that Gartner has received from clients has been consistently high. In both years, they have breached the 3,000 mark, with more than 800 inquiries a quarter. Inquiries have come from a very wide range of industries, and from organizations of varying levels of maturity.