How to Prevent Big Data Analytics Failures

A combination of factors usually derails big data implementations. Problems and failures occur due to factors including strategy, people, culture, capacities, inattention to analytics details or the nuances of implemented tools, all exacerbated by the rapid advancement of the digital economy.

Ahead of the Gartner Business Intelligence, Analytics and Information Management Summit in Sydney in February, Gartner research director Svetlana Sicular said that faced with overwhelming choices and new challenges, technical professionals often become distracted from the fundamental priorities — finding the right problems for big data solutions, questioning the data and understanding the nuances of the analytical models that are applied to big data.

“To succeed, you must develop a viable strategy to deliver business value from a big data initiative,” said Ms. Sicular. “Then map out and acquire or develop the missing and specialized skills that are needed. Once strategy and skill priorities are addressed, then you can move on to big data analytics.”

Learning from the pitfalls faced by others

We often hear of big data success stories, but some of the most revealing information that can help planners of big data analytics is information about failures. One of the most common failures involves setting overly optimistic expectations when a skilled team is not in place to deliver.

The CEO of a retail chain recognized that to stay competitive, his company needed a recommendation engine, like Amazon's famous "Customers who bought this item also bought …" feature. The retailer had never undertaken any big data projects before, but executives promised the CEO that an engine would be operational within six months.

Learn More: Drive business alignment and profits with the right strategy

IT worked hard to implement a collaborative-filtering algorithm that usually powers a recommendation engine, but struggled with the data sparsity and scale of huge datasets of purchased items and customer browsing history, as well as the available inventory that was constantly changing, which required further learning and additional skills.

“ Once strategy and skill priorities are addressed, then you can move on to big data analytics.”

To meet the deadline, the team created a fake recommendation engine with bed sheets as the recommended product, regardless of what was being purchased. While there weren’t any actual analytics behind it, it produced a significant sales lift. It then took two years from the CEO directive before they actually developed an effective engine. In that time, they had added behavioral experts, specialists in operations research and engineers into the team to stabilize the company's big data infrastructure.

This demonstrates that a key success factor for implementing big data analytics is the organization's ability to build, grow and sustain a multi-disciplinary team with the expertise needed to address identified business problems.

Organizations should also allow ample time to deliver meaningful results. Alternatively, they may consider buying a commercial recommendation engine. In this case, the strategy should include a decision on build versus buy.