The Missing Link to Make Big Data Go Mainstream

In Information Management (IM), there is a clash between two schools of thought. There is the established order of information managers, architects and so forth who really have a handle on IM. They know how to translate strategy and requirements into metrics, into data models and architecture, and how to acquire the information in an efficient manner.

However, with the rise of big data, another school of thought stood up. IM is about hunting for useful data, experimenting with it, and responding to an immediate business use. All kinds of software is used, data is shifted back and forth, and it’s easier to start the next exercise from scratch than investing in reuse.

Both schools of thought don’t show a lot of respect for each other. The “suits” see big data as hype, even as a threat. While the “hoodies” see big data as the next generation. Big data discovery is the next level of self-service analytics used to explore big data sources.

In their session on big data discovery at the Gartner Business Intelligence & Analytics Summit, Gartner analysts Dan Sommer and Joao Tapadinhas discussed how next-generation data discovery will change the market and competitive landscape.

“Big data discovery is a rising trend in information exploration and analytics, combining strengths and benefits from data discovery, advanced analytics and big data,” said Mr. Sommer. “Just like data discovery before, with traditional data sources and less advanced analytic capabilities, big data discovery will have a positive impact in the business analytics landscape. It will broaden the user base capable of leveraging big data sources to answer business questions, increasing the pervasiveness of big data analytics.”

There are potential roadblocks. A narrow set of offerings in the market and current limitations of tools and processes could hinder the deployment and create difficulties in operating a big data discovery environment. Lack of proper governance and failing to update existing processes will turn broad access to big data sources into a potential burden for organizations, where users may create inconsistent and often contradictory insights.

Convergence Leads to Big Data Discovery

The convergence of data discovery, advanced analytics and big data paves the way for big data discovery and will help business intelligence and analytics leaders start the exploration of big data sources. The blending in a single tool or tightly coupled portfolio, the ease of use, interactivity and agility of data discovery, with the richness of analysis and scale, diversity or immediacy of big data, will be the inception of big data discovery.

In essence, big data discovery will become to big data analytics, what data discovery is to advanced analytics, a set of tools that is:

• Simpler to use and accessible to a wide range of users
• Not capable of the same level of analytics complexity

When compared with data discovery, big data discovery represents an evolution in data manipulation sophistication.

• It has scalability, understanding new types of data sources and performance that allows it to explore data in high volumes, wide varieties or extreme velocities – big data sources.
• It will have a richer set of data manipulation features that require more skilled users.

“Big data discovery shouldn't be the endgame objective — it will not deliver the breadth and depth of big data analytics initiatives,” said Mr. Tapadinhas. “However, big data discovery will help advance the analytics maturity of the organization, will start training some of the future data scientists, can provide the first batch of insights that may raise awareness to new opportunities and may provide enough return on investment to justify the business case for big data analytics. It is the missing link that will make big data go mainstream.”