How to Prevent Big Data Analytics Failures

Big data analytics projects don’t fail for a single reason, nor due to technology alone.

A combination of factors usually derails big data implementations. Problems and failures occur due to factors including strategy, people, culture, capacities, inattention to analytics details or the nuances of implemented tools, all exacerbated by the rapid advancement of the digital economy.

Ahead of the Gartner Business Intelligence, Analytics and Information Management Summit in Sydney in February, Gartner research director Svetlana Sicular said that faced with overwhelming choices and new challenges, technical professionals often become distracted from the fundamental priorities — finding the right problems for big data solutions, questioning the data and understanding the nuances of the analytical models that are applied to big data.

“To succeed, you must develop a viable strategy to deliver business value from a big data initiative,” said Ms. Sicular. “Then map out and acquire or develop the missing and specialized skills that are needed. Once strategy and skill priorities are addressed, then you can move on to big data analytics.”

IT Leadership Vision for 2021

Emerging trends, challenges and next steps

Download eBook

Learning from the pitfalls faced by others

We often hear of big data success stories, but some of the most revealing information that can help planners of big data analytics is information about failures. One of the most common failures involves setting overly optimistic expectations when a skilled team is not in place to deliver.

The CEO of a retail chain recognized that to stay competitive, his company needed a recommendation engine, like Amazon’s famous “Customers who bought this item also bought …” feature. The retailer had never undertaken any big data projects before, but executives promised the CEO that an engine would be operational within six months.

Learn More: Drive business alignment and profits with the right strategy.

IT worked hard to implement a collaborative-filtering algorithm that usually powers a recommendation engine, but struggled with the data sparsity and scale of huge datasets of purchased items and customer browsing history, as well as the available inventory that was constantly changing, which required further learning and additional skills.

Once strategy and skill priorities are addressed, then you can move on to big data analytics.

To meet the deadline, the team created a fake recommendation engine with bed sheets as the recommended product, regardless of what was being purchased. While there weren’t any actual analytics behind it, it produced a significant sales lift. It then took two years from the CEO directive before they actually developed an effective engine. In that time, they had added behavioral experts, specialists in operations research and engineers into the team to stabilize the company’s big data infrastructure.

This demonstrates that a key success factor for implementing big data analytics is the organisation’s ability to build, grow and sustain a multi-disciplinary team with the expertise needed to address identified business problems.

Organisations should also allow ample time to deliver meaningful results. Alternatively, they may consider buying a commercial recommendation engine. In this case, the strategy should include a decision on build versus buy.

This is just one of many real-life examples organizations can learn from. Gartner clients can read more in the report: Big Data Analytics Failures and How to Prevent Them by Svetlana Sicular.

Get Smarter

Follow #Gartner

Attend a Gartner event

Explore Gartner Conferences

Gartner IT Roadmap for Cybersecurity: A Resilient Strategy

Gartner IT roadmap for cybersecurity based on unbiased research and...

Learn More


Get actionable advice in 60 minutes from the world's most respected experts. Keep pace with the latest issues that impact business.

Start Watching