Hope springs eternal, especially in Big Data. Despite widespread failure to achieve much of anything with Big Data projects, gargantuan piles of cash keep flowing into such projects, hitting $31 billion in 2013 and expected to top $114 billion by 2018.
Yet while 60% of executives believe Big Data will upend their industries within three years—according to a recent Capgemini report—a mere 8% describe their own projects as “very successful,” while another 27% call their efforts “successful.”
Given how much companies are spending, one would hope for better returns. Real success, however, derives from a cultural affinity for data, and not simply a technology purchase.
Failure All The Way Down
No one seems to dispute the inherent value of data, and the more of it the better. In Capgemini’s survey, fully 60% of respondents believe Big Data is going to change the world, starting with their industries.
Yet when asked about the status of their Big Data initiatives, it’s clear that reality bites:
For those that have been paying attention, this isn’t really news. After all, Gartner found a few years ago that while everyone was jumping into Big Data, few knew how to make it work.
As to why Big Data projects fail, the answer is “it depends.” Some of the reasons are cultural (“ineffective coordination of teams across the organization”), while others are more easily fixed (“dependency on legacy systems”):
Looking at this list, it’s hard to see how things improve in the short term.
Teaching Old Data Dogs New Data Tricks
Some vendors pitch a “data lake” as the solution to the first problem. Capgemini’s survey finds that 79% of enterprises haven’t completely integrated data sources from across the organization. To make this simpler, the data lake advocates insist that enterprises don’t need to standardize data as it enters the organization; instead they can keep it in its original format and just store it in one big repository.
See also: Oh, Go Jump In A Data Lake
While this sounds simple, it’s not clear that it’s actually useful.
I’ve heard some call such a data lake a “Hadump,” an unflattering play on the decentralized storage framework Hadoop that suggests just because all the data resides in one place doesn’t make it useful. As Gartner analyst Nick Huedecker has pointed out:
The fundamental issue with the data lake is that it makes certain assumptions about the users of information. It assumes that users recognize or understand the contextual bias of how data is captured, that they know how to merge and reconcile different data sources without ‘a priori knowledge’ and that they understand the incomplete nature of datasets, regardless of structure.
So some fail with newfangled, Hadoop-inspired data lakes, while many more fail by trying to get antiquated data infrastructure (e.g., relational databases) to fit modern data (messy, disparate, and lots of it).
The Cultural Problem
But the biggest cause of failure, even if not acknowledged as such, is that most enterprises simply don’t have a culture of data-centricity. At best, they treat “Big Data” as a discrete project with a definitive completion date.
As such, they’re not set up to succeed, as the Capgemini report finds:
There are many factors that go into the making of a successful Big Data implementation. However, the single biggest factor that we observed was that organizations that have a strong operating model stood apart. This operating model has multiple distinct elements, which include, among others, a well-defined organizational structure, systematic implementation plan, and strong leadership support.
Each of these three things ties into a corporate culture that appreciates and is built around data.
I would also add, following something that Zoomdata’s Justin Langseth recently said to me, that design is an essential element of any successful Big Data project. The best Big Data projects will bring data to life for the rank-and-file within an enterprise, not merely the high priests and priestesses of data science.
In sum, Big Data success flows from a cultural affinity for data, which can be sparked by a strong leader within an organization, but ultimately must become how an entire company thinks about its business.
Image courtesy of Shutterstock
