How to Prevent Big Data Analytics Failures

Big data analytics projects don’t fail for a single reason, nor due to technology alone.

A combination of factors usually derails big data implementations. Problems and failures occur due to factors including strategy, people, culture, capacities, inattention to analytics details or the nuances of implemented tools, all exacerbated by the rapid advancement of the digital economy.

Ahead of the Gartner Business Intelligence, Analytics and Information Management Summit in Sydney in February, Gartner research director Svetlana Sicular said that faced with overwhelming choices and new challenges, technical professionals often become distracted from the fundamental priorities — finding the right problems for big data solutions, questioning the data and understanding the nuances of the analytical models that are applied to big data.

“To succeed, you must develop a viable strategy to deliver business value from a big data initiative,” said Ms. Sicular. “Then map out and acquire or develop the missing and specialized skills that are needed. Once strategy and skill priorities are addressed, then you can move on to big data analytics.”

Design a Data and Analytics Strategy

Advance your organization's strategy by communicating the business value of data and analytics.

Download the eBook

Learning from the pitfalls faced by others

We often hear of big data success stories, but some of the most revealing information that can help planners of big data analytics is information about failures. One of the most common failures involves setting overly optimistic expectations when a skilled team is not in place to deliver.

The CEO of a retail chain recognized that to stay competitive, his company needed a recommendation engine, like Amazon’s famous “Customers who bought this item also bought …” feature. The retailer had never undertaken any big data projects before, but executives promised the CEO that an engine would be operational within six months.

Learn More: Drive business alignment and profits with the right strategy.

IT worked hard to implement a collaborative-filtering algorithm that usually powers a recommendation engine, but struggled with the data sparsity and scale of huge datasets of purchased items and customer browsing history, as well as the available inventory that was constantly changing, which required further learning and additional skills.

Once strategy and skill priorities are addressed, then you can move on to big data analytics.

To meet the deadline, the team created a fake recommendation engine with bed sheets as the recommended product, regardless of what was being purchased. While there weren’t any actual analytics behind it, it produced a significant sales lift. It then took two years from the CEO directive before they actually developed an effective engine. In that time, they had added behavioral experts, specialists in operations research and engineers into the team to stabilize the company’s big data infrastructure.

This demonstrates that a key success factor for implementing big data analytics is the organisation’s ability to build, grow and sustain a multi-disciplinary team with the expertise needed to address identified business problems.

Organisations should also allow ample time to deliver meaningful results. Alternatively, they may consider buying a commercial recommendation engine. In this case, the strategy should include a decision on build versus buy.

This is just one of many real-life examples organizations can learn from. Gartner clients can read more in the report: Big Data Analytics Failures and How to Prevent Them by Svetlana Sicular.

Get Smarter

Gartner Data & Analytics Summits

Get the tools and insights you need to build on the fundamentals of data and analytics.

Explore Gartner Conferences

Predicts 2019: Data and Analytics Strategy

Data and analytics are the key accelerants of digitalization, transformation and “ContinuousNext” efforts. As a result, data and analytics leaders will be counted upon to affect corporate strategy and value, change management, business ethics, and execution performance.

Read Free Gartner Research


Get actionable advice in 60 minutes from the world's most respected experts. Keep pace with the latest issues that impact business.

Start Watching