Share with your friends


Analytics Magazine

Dark analytics: Shedding light on a new business asset

By Nitin Mittal

Deep within the astonishing volumes of raw information generated by business transactions, social media, search engines, IoT and countless other sources, valuable intelligence about customers, markets and organizations, lies waiting to be discovered.

Leveraging advanced technologies to explore this expansive universe of unstructured and “dark” data reveals hidden insights to inform decision-making and chart new paths to the future. Machine learning, robotics process automation, visualization, natural language processing, and image and video analysis allow questions to be answered and opportunities brought to light that were unimaginable only a few years ago.

We’re only beginning to explore the digital universe, yet significant business value lies within. In fact, International Digital Corporation predicts that organizations that can analyze all relevant data and deliver actionable information could achieve an extra $430 billion in productivity gains over their peers by 2020 [1].

The influence of dark data is inescapable, increasingly driving innovation and value to customers and stakeholders. From financial services to healthcare, we’re starting to see the powerful impact of evolving technologies, new types of data and its potential to drive competitive advantage and shape entire industries. In the automotive industry, connected cars could help make usage-based insurance possible and accurately predict when vehicles need maintenance. Some automakers are beginning to experiment with an augmented reality dashboard.

Defining Dark Data

With a digital universe expected to reach 44 zetabytes by 2020, and 90 percent of it being unstructured data from the IoT and non-traditional sources [2], how is dark data defined? Today’s dark analytics efforts usually focus on three dimensions:

  1. Untapped data already in your possession. Most organizations have large collections of structured and unstructured data sitting idle. On the structured side, it’s often due to the difficulty of making connections between disparate data sets. Insights lie waiting to be discovered: One company mapped the addresses of employees against workplace satisfaction ratings and retention data, finding that one of the biggest factors fueling voluntary turnover was commute time.

Unstructured data is often text-based, and until recently, tools and techniques needed to leverage them efficiently did not exist. Today, scanned patient records could hold the key to better understanding disease and prescribing effective treatments; executive emails and communications could unearth wisdom needed to pass along to a younger generation of workers.

  1. Nontraditional unstructured data. Another dark analytics dimension focuses on data such as audio and video files and still images that could not be explored until now. Now, computer vision, advanced pattern recognition, and video and sound analytics allow companies to mine this data to better understand customers, employees, operations and markets.

This insight can be illuminating; retailers have a window into customer sentiment by analyzing in-store posture and facial expressions or online browsing patterns. Oil and gas companies can use acoustic sensors to monitor pipelines and algorithms to provide visibility into flow rates and fluid composition.

  1. Data in the deep web. The deep web may offer the largest of body of untapped information – data curated by academics, government agencies, communities and other third-party domains – that is often hidden behind firewalls. Organizations may soon be able to curate competitive intelligence using search tools designed to help target specific data types. Analysis of this “deep web” data is especially promising in its potential to allow better prevention, detection and response to cyber threats.

Light Up the Dark Side

Developing a strategy for discovering value in unstructured data can help your organization generate insights today and prepare for even greater opportunities in the years ahead. How can your organization get the most value out of the mountains of data that it has created, owns or has access to? To help optimize the value of this business asset, consider these practical steps:

Ask the right questions. Work with business teams to identify specific questions that need to be answered. Then identify the sources of data that make the most sense for your analytics efforts. To boost sales of sports equipment, analytics teams can focus on sales transactions, inventory and product pricing in a specific geographic area. Supplementing the data with unstructured data, such as instore analysis of foot traffic or social media trends, generates more nuanced insights.

Look outside of your organization. Augmenting your data with publicly available demographic, location and statistical information helps put insights in context. For example, when a physician makes recommendations to an asthma patient about how to manage symptoms, he can also provide short-term solutions to help her deal with flare-ups during pollen season.

Expand analytics talent for impact. Optimizing dark data analytics depends on assembling the right teams to meet specific needs and identify emerging opportunities. Build a cohesive team that encompasses organizational, business and technical knowledge.

View analytics as a business-driven effort. Plan your project with your organization’s business goals in mind to determine the value that must be delivered, define the questions to ask and decide how to harness data to generate the right answers. Data analytics then becomes an insight-driven advantage that wins support throughout the organization and fuels future endeavors.

Think broadly. As new strategies and capabilities for using dark analytics are developed, consider how they can be extended across the organization, as well as to customers, vendors and business partners.

By tapping into dark data, organizations have an opportunity to turn previously hidden or unknown patterns and correlations into powerful insights. Uncovering this business value leads to new opportunities, reduced risk and increased analytics ROI.

The potential is exciting, but staying grounded with specific business questions that have a defined scope and measurable value is critical. Focus your dark analytics efforts on areas that matter to your business – and avoid getting lost in the increasingly vast unknown.

Nitin Mittal is a principal in Deloitte Consulting LLP and leads Deloitte Consulting’s Analytics & Information Management practice. His team works with companies from many industries to fully leverage the potential of analytics and emerging technologies.


  1. “IDC FutureScape: Worldwide big data, business analytics, and cognitive software predictions,” International Data Corporation, 2016.
  2. EMC Digital Universe with research and analysis by IDC, “The digital universe of opportunities: Rich data and the increasing value of the Internet of Things,” April 2014,

Related Posts

  • 82
    Deep within the astonishing volumes of raw information generated by business transactions, social media, search engines, IoT and countless other sources, valuable intelligence about customers, markets and organizations, lies waiting to be discovered. Leveraging advanced technologies to explore this expansive universe of unstructured and “dark” data reveals hidden insights to…
    Tags: data, analytics, business, dark
  • 75
    Features Shedding light on dark analytics New business asset: Leveraging advanced technologies to explore unstructured and “dark” data reveals hidden insights. By Nitin Mittal Dark data: Two sides of same coin Lost opportunity and security risk: Dark data can be tapped to generate more opportunities or remain in the dark,…
    Tags: data, dark, analytics
  • 70
    Content/Interactive Marketing Opportunities can help you build a successful content marketing program or interactive lead generation program. Enhance your position as an industry thought leader and expert in the analytics profession by promoting the following content formats on Product Videos Software Demonstrations White Papers Case Studiesa Research Reports…
    Tags: analytics, data, analysis
  • 69
    The CUNY School of Professional Studies is offering a new online master of science degree in data analytics. The program prepares its graduates for high-demand and fast-growing careers as data analysts, data specialists, business intelligence analysts, information analysts and data engineers in such fields as business, operations, marketing, social media,…
    Tags: data, analytics, business
  • 68
    Benjamin Franklin offered this sage advice in the 18th century, but he left one key question unanswered: How? How do you successfully drive a business? More specifically, how do you develop the business strategy drivers that incite a business to grow and thrive? The 21st-century solution has proven to be…
    Tags: data, analytics, business


Artificial intelligence a game changer for personal devices

Emotion artificial intelligence (AI) systems are becoming so sophisticated that Gartner, Inc. predicts that by 2022, personal devices will know more about an individual’s emotional state than his or her own family. AI is generating multiple disruptive forces that are reshaping the way we interact with personal technologies. Read more →

FICO predicts AI and blockchain will meet in 2018

Scott Zoldi, chief analytics officer at FICO, in his AI predictions for 2018, predicts the rise of “defensive AI” and manipulative chatbots. Blockchain will use AI to search through relationship data, Zoldi says, and defensive AI will be used to protect systems from malicious AI and machine learning. Adds Zoldi: Chatbots will get so good at understanding us they will learn how to manipulate us. Read more →



2018 INFORMS Conference on Business Analytics and Operations Research
April 15-17, 2018, Baltimore


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to