Share with your friends


Analytics Magazine

Dark analytics: Shedding light on a new business asset

By Nitin Mittal

Deep within the astonishing volumes of raw information generated by business transactions, social media, search engines, IoT and countless other sources, valuable intelligence about customers, markets and organizations, lies waiting to be discovered.

Leveraging advanced technologies to explore this expansive universe of unstructured and “dark” data reveals hidden insights to inform decision-making and chart new paths to the future. Machine learning, robotics process automation, visualization, natural language processing, and image and video analysis allow questions to be answered and opportunities brought to light that were unimaginable only a few years ago.

We’re only beginning to explore the digital universe, yet significant business value lies within. In fact, International Digital Corporation predicts that organizations that can analyze all relevant data and deliver actionable information could achieve an extra $430 billion in productivity gains over their peers by 2020 [1].

The influence of dark data is inescapable, increasingly driving innovation and value to customers and stakeholders. From financial services to healthcare, we’re starting to see the powerful impact of evolving technologies, new types of data and its potential to drive competitive advantage and shape entire industries. In the automotive industry, connected cars could help make usage-based insurance possible and accurately predict when vehicles need maintenance. Some automakers are beginning to experiment with an augmented reality dashboard.

Defining Dark Data

With a digital universe expected to reach 44 zetabytes by 2020, and 90 percent of it being unstructured data from the IoT and non-traditional sources [2], how is dark data defined? Today’s dark analytics efforts usually focus on three dimensions:

  1. Untapped data already in your possession. Most organizations have large collections of structured and unstructured data sitting idle. On the structured side, it’s often due to the difficulty of making connections between disparate data sets. Insights lie waiting to be discovered: One company mapped the addresses of employees against workplace satisfaction ratings and retention data, finding that one of the biggest factors fueling voluntary turnover was commute time.

Unstructured data is often text-based, and until recently, tools and techniques needed to leverage them efficiently did not exist. Today, scanned patient records could hold the key to better understanding disease and prescribing effective treatments; executive emails and communications could unearth wisdom needed to pass along to a younger generation of workers.

  1. Nontraditional unstructured data. Another dark analytics dimension focuses on data such as audio and video files and still images that could not be explored until now. Now, computer vision, advanced pattern recognition, and video and sound analytics allow companies to mine this data to better understand customers, employees, operations and markets.

This insight can be illuminating; retailers have a window into customer sentiment by analyzing in-store posture and facial expressions or online browsing patterns. Oil and gas companies can use acoustic sensors to monitor pipelines and algorithms to provide visibility into flow rates and fluid composition.

  1. Data in the deep web. The deep web may offer the largest of body of untapped information – data curated by academics, government agencies, communities and other third-party domains – that is often hidden behind firewalls. Organizations may soon be able to curate competitive intelligence using search tools designed to help target specific data types. Analysis of this “deep web” data is especially promising in its potential to allow better prevention, detection and response to cyber threats.

Light Up the Dark Side

Developing a strategy for discovering value in unstructured data can help your organization generate insights today and prepare for even greater opportunities in the years ahead. How can your organization get the most value out of the mountains of data that it has created, owns or has access to? To help optimize the value of this business asset, consider these practical steps:

Ask the right questions. Work with business teams to identify specific questions that need to be answered. Then identify the sources of data that make the most sense for your analytics efforts. To boost sales of sports equipment, analytics teams can focus on sales transactions, inventory and product pricing in a specific geographic area. Supplementing the data with unstructured data, such as instore analysis of foot traffic or social media trends, generates more nuanced insights.

Look outside of your organization. Augmenting your data with publicly available demographic, location and statistical information helps put insights in context. For example, when a physician makes recommendations to an asthma patient about how to manage symptoms, he can also provide short-term solutions to help her deal with flare-ups during pollen season.

Expand analytics talent for impact. Optimizing dark data analytics depends on assembling the right teams to meet specific needs and identify emerging opportunities. Build a cohesive team that encompasses organizational, business and technical knowledge.

View analytics as a business-driven effort. Plan your project with your organization’s business goals in mind to determine the value that must be delivered, define the questions to ask and decide how to harness data to generate the right answers. Data analytics then becomes an insight-driven advantage that wins support throughout the organization and fuels future endeavors.

Think broadly. As new strategies and capabilities for using dark analytics are developed, consider how they can be extended across the organization, as well as to customers, vendors and business partners.

By tapping into dark data, organizations have an opportunity to turn previously hidden or unknown patterns and correlations into powerful insights. Uncovering this business value leads to new opportunities, reduced risk and increased analytics ROI.

The potential is exciting, but staying grounded with specific business questions that have a defined scope and measurable value is critical. Focus your dark analytics efforts on areas that matter to your business – and avoid getting lost in the increasingly vast unknown.

Nitin Mittal is a principal in Deloitte Consulting LLP and leads Deloitte Consulting’s Analytics & Information Management practice. His team works with companies from many industries to fully leverage the potential of analytics and emerging technologies.


  1. “IDC FutureScape: Worldwide big data, business analytics, and cognitive software predictions,” International Data Corporation, 2016.
  2. EMC Digital Universe with research and analysis by IDC, “The digital universe of opportunities: Rich data and the increasing value of the Internet of Things,” April 2014,

Related Posts

  • 82
    Deep within the astonishing volumes of raw information generated by business transactions, social media, search engines, IoT and countless other sources, valuable intelligence about customers, markets and organizations, lies waiting to be discovered. Leveraging advanced technologies to explore this expansive universe of unstructured and “dark” data reveals hidden insights to…
    Tags: data, analytics, business, dark
  • 75
    Features Shedding light on dark analytics New business asset: Leveraging advanced technologies to explore unstructured and “dark” data reveals hidden insights. By Nitin Mittal Dark data: Two sides of same coin Lost opportunity and security risk: Dark data can be tapped to generate more opportunities or remain in the dark,…
    Tags: data, dark, analytics
  • 70
    Content/Interactive Marketing Opportunities can help you build a successful content marketing program or interactive lead generation program. Enhance your position as an industry thought leader and expert in the analytics profession by promoting the following content formats on Product Videos Software Demonstrations White Papers Case Studiesa Research Reports…
    Tags: analytics, data, analysis
  • 69
    The CUNY School of Professional Studies is offering a new online master of science degree in data analytics. The program prepares its graduates for high-demand and fast-growing careers as data analysts, data specialists, business intelligence analysts, information analysts and data engineers in such fields as business, operations, marketing, social media,…
    Tags: data, analytics, business
  • 68
    Benjamin Franklin offered this sage advice in the 18th century, but he left one key question unanswered: How? How do you successfully drive a business? More specifically, how do you develop the business strategy drivers that incite a business to grow and thrive? The 21st-century solution has proven to be…
    Tags: data, analytics, business


Using machine learning and optimization to improve refugee integration

Andrew C. Trapp, a professor at the Foisie Business School at Worcester Polytechnic Institute (WPI), received a $320,000 National Science Foundation (NSF) grant to develop a computational tool to help humanitarian aid organizations significantly improve refugees’ chances of successfully resettling and integrating into a new country. Built upon ongoing work with an international team of computer scientists and economists, the tool integrates machine learning and optimization algorithms, along with complex computation of data, to match refugees to communities where they will find appropriate resources, including employment opportunities. Read more →

Gartner releases Healthcare Supply Chain Top 25 rankings

Gartner, Inc. has released its 10th annual Healthcare Supply Chain Top 25 ranking. The rankings recognize organizations across the healthcare value chain that demonstrate leadership in improving human life at sustainable costs. “Healthcare supply chains today face a multitude of challenges: increasing cost pressures and patient expectations, as well as the need to keep up with rapid technology advancement, to name just a few,” says Stephen Meyer, senior director at Gartner. Read more →

Meet CIMON, the first AI-powered astronaut assistant

CIMON, the world’s first artificial intelligence-enabled astronaut assistant, made its debut aboard the International Space Station. The ISS’s newest crew member, developed and built in Germany, was called into action on Nov. 15 with the command, “Wake up, CIMON!,” by German ESA astronaut Alexander Gerst, who has been living and working on the ISS since June 8. Read more →



INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.


Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to