Survey Analysis: Big Data Adoption in 2013 Shows Substance Behind the Hype

12 September 2013 ID:G00255160
Analyst(s): Lisa Kart, Nick Heudecker, Frank Buytendijk

VIEW SUMMARY

In Gartner's 2013 big data study, information and business leaders most often associate the term "opportunity" with "big data." This positive attitude translates to increased investments in and adoption of big data technology.

Overview

Key Findings

  • Big data investments in 2013 continue to rise, with 64% of organizations investing or planning to invest in big data technology compared with 58% last year. Investments are led by media and communications, banking and services. Planned investments the next two years are highest for transportation, healthcare and insurance. Fewer than 8% of respondents have deployed.
  • Enhanced customer experience is the top big data priority, with process efficiency close behind. Organizations struggle most with knowing how to get value from big data, compared with last year's top challenge of governance. Obtaining skills remains a critical issue for one-third of organizations.
  • Big Data is touted as being about unconventional data sources and the use of new and innovative technologies; this is not yet reflected in the chosen sources for first projects. Transaction and log data still dominates the big data being analyzed.
  • Big data technologies supplement — but do not replace — existing information management and analytics. As a result, cloud adoption, with its supplementary nature, is the overriding technology that companies are using to derive value from big data.

Recommendations

  • Assess your big data strategy, adoption and priorities relative to industry peers, but explore outside your industry for innovative use cases and applications.
  • Align big data initiatives to business insights and decisions. Prioritize business problems and the data sources that may bring deeper insights to those opportunities.
  • Seek experienced providers when you need help overcoming hurdles around skills, technology, strategy or business objectives. Invest in data science skills and new technology skills.

Table of Contents

Survey Objective

This document was revised on 13 November 2013. The document you are viewing is the corrected version. For more information, see the Corrections page on gartner.com.

The hype around "big data" continues to drive increased investment and attention. Organizations across industries and geographies see "opportunity" and real business value more than the "smoke and mirrors" with which hypes usually come. This is the general picture that emerges from a June 2013 Gartner Research Circle survey on big data.

The Gartner Research Circle is a Gartner-managed panel composed of IT and business leaders. It includes global organizations across all industries, both Gartner clients and nonclients. In total, 720 Research Circle members participated by indicating their organization's investment plans around technology to support big data (see the Methodology section for more details).

The goal of the survey was to examine organizations' technology investment plans around big data, stages of big data adoption, business problems solved, data, technology and challenges. Next, compare these results with the responses in last year's survey (conducted in June 2012).

Data Insights

There Is Substance Beyond the Hype

"Big data" (see "Hype Cycle for Big Data, 2013") is one of the most hyped technology terms at the moment. Clients ask Gartner how much is vendor marketing versus real uptake and investment by organizations in their industry. The investment numbers across industries are impressive and point to big data planning and investment activities beyond the hype.

Big data investments in 2013 continue to rise from 2012, with 64% of organizations investing or planning to invest in big data technology compared to 58% last year (see Figure 1). To detail the 64%, 30% have already invested in big data technology, 19% plan to invest within the next year, and an additional 15% plan to invest within two years. Clearly, not all who said they planned to invest have invested, but some progress has been made. Fewer organizations (only 5% compared with 11% last year) don't know what their big data plans are.

Industries leading big data investments in 2013 are media and communications, banking and services. This is a change from last year's leaders — education, healthcare and transportation. Planned investments over the next two years are highest for transportation, healthcare and insurance (see Figure 2). However, every vertical industry again shows big data investment and planned investment.

North America continues to lead the investments, with Asia/Pacific notably ambitious in its plans to invest during the next two years (see Figure 3). Consistent with Gartner experience, EMEA and Latin America tend to lag technology adoption, for which big data is no different.

Figure 1. Big Data Investments on the Rise
Figure 1.Big Data Investments on the Rise

Source: Gartner (September 2013)

Figure 2. Big Data Investment by Industry
Figure 2.Big Data Investment by Industry

Source: Gartner (September 2013)

Figure 3. Big Data Investment by Region
Figure 3.Big Data Investment by Region

Source: Gartner (September 2013)

Investment typically has different stages that organizations go through. It starts with knowledge gathering, followed by strategy setting. The investment is small, and mostly consists of time. Then it is typically followed by an experiment or proof of concept. Still, the investment is small and tentative. Then, after completing a successful pilot, the first deployments take place. Here the investment curve rises. Over time business operations start to rely on the deployments, the investments move from implementing systems to managing them.

For big data, 2013 is the year of experimentation and early deployment. The adoption is still at the early stages with fewer than 8% of all respondents indicating their organization has deployed big data solutions. Twenty percent are piloting and experimenting, 18% are developing a strategy, 19% are knowledge gathering, and the remainder have no plans or don't know.

When we look just at the adoption for those who have made investments, 70% of organizations have moved past the early knowledge gathering and strategy formation phases and into piloting (44%) and deployment (25%). Among those planning to invest over the next two years, 80% are in the earlier stages (knowledge gathering and strategy phase) (see Figure 4).

Figure 4. Big Data Adoption by Investment Stage
Figure 4.Big Data Adoption by Investment Stage

Source: Gartner (September 2013)

The industries furthest along the adoption curve are banking (13% deployed) and manufacturing and natural resources (11% deployed). To get ready for big data, most organizations (71%) are determining the business need and building the business case. Other areas include determining the skills needed, ensuring data "trust," acquiring tools to process big data, and building infrastructure. Interactions with Gartner clients indicate skill and talent acquisition is the primary concern for big data projects. Infrastructure and deployment options, such as whether to deploy in the cloud or on-premises, are also a concern.

Gartner expects the interest level and investments in big data to continue, with those at the earlier stages of adoption to move to piloting and deploying solutions. However, several technologies associated with big data have yet to mature past the Trough of Disillusionment in the Hype Cycle. This indicates companies may encounter implementation difficulties because of vendor immaturity and lack of consistent standards and best practices.

Priorities and Challenges Are Changing

There is a difference between the issues that respondents feel they can address with big data, and the priority they give them. For instance, companies may have a long wish list of areas they would like to address, but they will prioritize a shorter list or place more weight on specific opportunities. Factors in prioritization include time and budget constraints, strategic alignment of initiatives, or a certain required order. For instance, before you can invest in big data analytics to improve customer intimacy, it may first be needed to improve certain internal business operations to get to a certain quality level.

Clearly, a range of business problems are being addressed using big data (see "Toolkit: Big Data Business Opportunities From Over 100 Use Cases"), although there are some clear patterns. In both of Gartner's 2012 and 2013 studies, business cases that improve process efficiency ("operational excellence") and business cases around customer experience dominate big data wish lists (see Figure 5).

Figure 5. Business Problems Addressed by Big Data
Figure 5.Business Problems Addressed by Big Data

N = 465; multiple responses allowed

Source: Gartner (September 2013)

Some big data activities are incremental to current business practices; for example, better understanding of customer needs, making processes more efficient, further reducing costs, or better detecting risks. These make up the majority of the use cases that Gartner sees today. Some organizations are engaging in more game-changing activities: 42% are developing new products and business models, and 23% are monetizing information directly. This is encouraging, as the big opportunities lie mostly in these areas.

Although there are many areas companies would like to address, a slightly different picture emerges when we ask about the priority of these categories: 54% rank enhanced customer experience as one of their top three priorities (see Figure 6). When looking at priorities, we must consider the respondent's industry. Industries that are driving the customer experience priority are retail, insurance, media and communications, and banking. Process efficiency is a top priority for 42% of organizations, driven by manufacturing, government, education, healthcare and transportation.

Creating new products and business models is a game changer for many. For the services industry, creating new products and business models using big data is their No. 1 priority.

Figure 6. Top Priorities for Big Data
Figure 6.Top Priorities for Big Data

N = 687

Source: Gartner (September 2013)

Big data still has its challenges, although more organizations are acknowledging the issues, and some shifts have occurred from last year. The top big data challenge across industries for 2013 is how to get value from big data (see Figure 7): 56% list it as a top struggle. This comes from not knowing what data sources to analyze, what business problems to solve, and getting stuck trying to make the business case.

To move up the adoption curve and ultimately get to deployment, IT leaders should ensure big data initiatives are tied to organizational goals and processes, and demonstrate the insights and value that these initiatives can bring to the business. Gartner sees many organizations getting stuck in the experimentation phase — finding interesting insights from new data sources, but often not connecting them to business processes.

Outside of technology and data concerns, some Gartner inquiry clients cite organizational and leadership problems in extracting value from big data projects. At its core, the problem is that executive leadership is often unaccustomed to making data-led decisions. They prefer to go with past experience or instinct.

Figure 7. Top Big Data Challenges
Figure 7.Top Big Data Challenges

N = 687 (excludes "don't know" responses)

Source: Gartner (September 2013)

Defining a strategy and obtaining the skills and capabilities needed remain a top challenge. In 2012, data governance issues (including security, privacy and data quality) were the top concerns, followed by how to get value from big data and skills needed. Although these boundary conditions seem to be less of an issue, it doesn't mean that are totally addressed. Gartner expects demand for skills to significantly outweigh the supply (see Note 1). This means organizations have a better grip on and understanding of the issues at hand. Gartner has observed that big data challenges shift with organizational maturity in information management, especially handling big data.

The top challenge for 15% of organizations is understanding what big data is. Most of this comes from respondents with "no plans to invest." Organizations should be educated about big data opportunities in their industry to ensure they are not missing the boat.

Conventional Data Sources Still Reign

Big data is often described as being about unconventional data sources and the use of new and innovative technologies. Despite a fair amount of hype around unstructured data and content such as video or image analytics, this is not yet reflected in the chosen sources for first projects.

Figure 8 shows that transactional sources are the dominant data types analyzed in big data initiatives at 70%, followed by log data at 55%. Nontraditional data sources, such as images, text and audio, have yet to gain much traction in big data initiatives. However, data from machines and sensors, emails and documents, and social media are new sources that are gaining traction and adding variety to the dimension of data.

Figure 8. Types of Data Analyzed
Figure 8.Types of Data Analyzed

N =465 (multiple responses allowed)

Source: Gartner (September 2013)

A clearer picture of analyzed data types emerges when the responses are separated by industry. Retail represents the largest processor of transactional data at 93%, but it leads all other industries in its processing of social media data at 73%. Combining these two data types supports the priority for enhanced customer experience. Healthcare's primary objective of improving process efficiency is addressed by leveraging machine data with one of the most opaque data sources — handwritten notes (see Figure 9).

An emerging source of big data is generated by machines and sensors, used most heavily by manufacturing and natural resources, transportation and insurance, for use cases ranging from preventative maintenance to telematics, the use of in-car devices that measure vehicle usage and maintenance requirements. Most industries are not broadly processing data from what Gartner calls the Internet of Things (see "The Internet of Things is Moving to the Mainstream"). However, inclusion of these data types is likely to increase as big data efforts mature and machine and sensor data becomes more prevalent.

Figure 9. Types of Big Data Analyzed by Industry
Figure 9.Types of Big Data Analyzed by Industry

Note: Highlighted cells indicate the top three data types by industry.
Multiple responses allowed

Source: Gartner (September 2013)

Gartner's client interactions indicate many organizations are focused on the volume aspects of big data, often overlooking the variety of data sources or the different velocities at which data is consumed. Many clients, more comfortable with traditional transactional or log-based data sources, often have difficulty grappling with how to integrate several data sources into big data analytics. Organizations have an easier time understanding how to deal with differing data velocities from a technical perspective, but often lack the operational capabilities to respond to the quickly changing conditions.

What we are likely observing is early progress tackling the relatable aspect of big data and volume, with additional aspects included as organizations become more mature. Interactions with clients indicate few have considered including external data sources to enrich analysis, and fewer still have considered data velocity. Many organizations find the variety dimension of big data a much bigger challenge than the volume or velocity dimensions (see Note 2). At the same time, many clients are looking to variety for new sources of information, competitive differentiation, and more value from their analytics. In the early stages of big data initiatives, progress is being made on higher volumes traditional data, while overcoming the challenges with velocity and the variety of new data sources.

Big Data Doesn't Replace Traditional Data and Analytics

As a relatively new set of technologies, one might expect to see Hadoop, MapReduce and NoSQL databases dominate the landscape; however, big data technologies are not really replacing incumbents such as business intelligence, relational database management systems and enterprise data warehouses. Instead, they supplement traditional information management and analytics. Cloud, also a supplementary technology to traditional IT approaches, is a large component of big data initiatives. Because the business is in the lead, cloud is seen as a way to address some of the issues related to cost and deployment speed.

When asked about big data technology use, respondents predominantly chose cloud computing at 41%, followed by search-based indexes at 27% (see Figure 10). The elastic-scaling requirements typical to big data projects makes the choice of cloud computing understandable, while data discovery needs are reinforced by the position of search-based indexes.

MapReduce & Alternatives, In-Memory DBMS, NoSQL DBMS and Complex Event Processing tied at 22%. Companies either are or will be looking at a variety of technologies to extract value from their big data projects, basing choices on the types of data processed, responsiveness and timeliness requirements, and analytical workload (see "Applying the Big Data Ecosystem").

Figure 10. Technologies Used to Derive Value From Big Data
Figure 10.Technologies Used to Derive Value From Big Data

N = 465 (multiple responses allowed)

Source: Gartner (September 2013)

Clients at the beginning phases of big data projects often have difficulty understanding which technologies are needed and how they differ. This may explain the groups of technologies at 22% and 15% to 16%. Clients have heard about the technologies, but aren't sure how they apply and the benefits and drawbacks of each.

Methodology

This study was conducted 3 through 14 June 2013 among Gartner Research Circle Members — a Gartner-managed panel composed of IT and business leaders.

Interviews were conducted online. Initial invitation to participate and two reminders were sent to 4,916 Research Circle members. In total, 720 Research Circle members participated by indicating their organization's investment plans around technology to support big data.

The respondent profiles are shown in Figures 11 and 12.

Figure 11. Respondent Profile — Company Characteristics
Figure 11.Respondent Profile — Company Characteristics

Source: Gartner (September 2013)

Figure 12. Respondent Profile — Region
Figure 12.Respondent Profile — Region

Source: Gartner (September 2013)

Vertical Industry Groupings

Gartner uses the vertical industry groupings outlined in Table 1.

Table 1. Vertical Industry Groupings

Vertical Industry

SIC Code

Manufacturing and Natural Resources

4, 16-21, 25

Communications

32

Media

22-24

Services

30, 31

Government

5-7

Education

2, 3

Retail

26-29

Wholesale Trade

40

Banking

1, 15

Insurance

11-14

Healthcare Providers

8-10

Transportation

33-37

Utilities

38, 39

Others

41

Source: Gartner (September 2013)

Detailed Industry Categories

Detailed industry categories are listed in Table 2.

Table 2. Detailed Industry Categories

Detailed Industry Categories

1

Banking

2

Education: higher education

3

Education: primary and secondary education

4

Energy resources and processing

5

Government: defense and intelligence

6

Government: local or regional

7

Government: national or international government

8

Healthcare providers: ambulatory clinic

9

Healthcare providers: hospital or integrated delivery network (IDN)

10

Healthcare providers: physician practice

11

Insurance: health Insurance (payer)

12

Insurance: life insurance

13

Insurance: property and casualty insurance

14

Insurance: others

15

Investment services

16

Manufacturing: automotive

17

Manufacturing: consumer nondurable products

18

Manufacturing: heavy industry

19

Manufacturing: IT hardware

20

Manufacturing: life sciences (pharma and biotech) or healthcare products (equipment and supplies)

21

Manufacturing: other

22

Media: broadcasting or cable

23

Media: entertainment (to include cultural institutions such as museums, etc.)

24

Media: publishing or advertising

25

Natural resources or materials

26

Retail: general retailers

27

Retail: grocery

28

Retail: restaurants and hotels

29

Retail: specialty retailers

30

Services: IT services and software

31

Services: other business, consulting or consumer services

32

Telecommunications

33

Transportation: air transport

34

Transportation: motor freight

35

Transportation: pipelines

36

Transportation: rail and water

37

Transportation: warehousing, couriers, support services

38

Utilities: electric or gas utilities

39

Utilities: water utilities

40

Wholesale durable and nondurable goods

41

All other

Source: Gartner (September 2013)

Definitions

Respondents were presented with Gartner's definition of big data: "Big data" is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.

Evidence

The fact base presented in this research comes from the Gartner Research Circle big data study conducted in June 2013, with 720 respondents. This study was a refresh and an expansion of the big data study run in July 2012. Additional insights come from client conversations, inquiry and so forth.

Note 1
Demand for Big Data Skills

Gartner predicts that by 2015, big data demand will reach 4.4 million jobs globally, but only one-third of those jobs will be filled (see "Gartner's Top Predictions for IT Organizations and Users, 2013 and Beyond: Balancing Economics, Risk, Opportunity and Innovation").

Note 2
Variety Is the Biggest Challenge

Organizations struggle more with variety than volume or velocity of data. When asked about the dimensions of data organizations struggle with most, 49% answered variety, while 35% answered volume and 16% velocity.