How to Tackle Dark Data

Most of us are guilty of “data hoarding”. Without a thought, we save every digital photo, email, document, presentation and spreadsheet, losing track of what we have saved along the way.

Across the enterprise, employees are blindly building a bottomless lake of data, and, in many cases, a corporate mantra of "save everything, just in case" is encouraging the behavior.

Email, instant messages, documents, ZIP files, log files, archived web content, partially developed and then abandoned applications, code snippets… all of this is now termed “dark data”.

Gartner defines dark data as “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.” It includes all data objects and types that have yet to be analyzed for any business or competitive intelligence or aid in business decision making.

“Increased data growth over the past decade has created an unstructured data nightmare,” says Alan Dayley, research director at Gartner. “It’s not just the cost to store it. Huge volumes of dark data make it harder to find what is useful and may mean we miss business opportunities.”

Gartner predicts that through 2021, more than 80% of organizations will fail to develop a consolidated data security policy across silos, leading to potential noncompliance, security breaches and financial liabilities.

To effectively manage data growth and security, information managers will need to deploy the right tools, and educate employees on how to overcome instinctual data hoarding.

The dark data opportunity

Operational data that is left unanalyzed can now be used as an economic opportunity for companies. They can look at using this data to drive new revenues or reduce internal costs.

Some examples of data that is often left dark include server log files that can give clues to website visitor behavior, customer call detail records that can indicate consumer sentiment and mobile geolocation data that can reveal traffic patterns to aid in business planning.

“No matter which types of dark data your organization collects, or how it is stored, the key to keeping data out of the dark is to ensure that you have a means of translating it from one form to another and ingesting it easily into whichever analytics platform you use,” says Dayley.

Generating large sums of data that serve nothing is useless knowledge. Whoever unlocks the reams of data and uses it strategically will win. Dayley’s recommendations for organizations to manage dark data are:

1. Start today. This is only going to get worse — don't wait for that unsavory catalyst.
2. Reach out to all stakeholders and then trim involvement of unnecessary but interested parties.
3. Take action — Move the data, secure the data, create accessibility of the data or  delete the data, depending on the desired business outcome.