The Data Economy: Why do so many analytics projects fail?
Key considerations for deep analytics on big data, learning and insights.
By (l-r) Haluk Demirkan and Bulent Dal
What is big data? Big data, which means many things to many people, is not a new technological fad. In addition to providing innovative solutions and operational insights to enduring challenges and opportunities, big data with deep analytics instigate new ways to transform processes, organizations, entire industries and even society. Pushing the boundaries of deep data analytics uncovers new insights and opportunities, and “big” depends on where you start and how you proceed.
Big data is not just “big.” The exponentially growing volume of data is only one of many characteristics that are often associated with big data, such as variety, velocity, veracity and others (the six Vs; see box).
According to Gartner Research, the worldwide market for analytics will remain the top focus for CIOs through 2017 . According to Gartner, more than half of all analytics projects fail because they aren’t completed within budget or on schedule, or because they fail to deliver the features and benefits that are optimistically agreed on at their outset.
The six Vs of big data
|• Volume (data at rest): terabytes to exabytes, petabytes to zettabytes of lots of data
• Velocity (data in motion): streaming data, milliseconds to seconds, how fast data is being produced and how fast the data must be processed to meet the need or demand
• Variety (data in many forms): structured, unstructured, text, multimedia, video, audio, sensor data, meter data, html, text, e-mails, etc.
• Veracity (data in doubt): uncertainty due to data inconsistency and incompleteness, ambiguities, latency, deception, model approximations, accuracy, quality, truthfulness or trustworthiness
• Variability (data in change): the differing ways in which the data may be interpreted; different questions require different interpretations
• Value (data for co-creation and deep learning): The relative importance of different complex data from distributed locations. Big data with deep analytics means greater insight and better decisions, something that every organization needs.
Today, an abundance of knowledge and experience exists to have successful data and analytics-enabled decision support systems. So why do so many of these projects fail, and why are so many executives and users still so unhappy? While there are many reasons for the high failure rate, the biggest reason is that companies still treat these projects as just another IT project. Big data analytics is neither a product nor a computer system. Instead, it should be considered a constantly evolving strategy, vision and architecture that continuously seeks to align an organization’s operations and direction with its strategic business goals and tactical and operational decisions. Table 1 includes a list of common mistakes that can doom analytics projects.
Key Considerations for Deep Analytics
We live in an era of big data. Whether you work in financial services, consumer goods, travel, transportation, healthcare, education, supply chain, logistics or industrial products and professional services, analytics are becoming a competitive necessity for your organization. But having big data – and even people who can manipulate it successfully – is not enough. Companies need managers who can partner effectively with analysts to ensure that their work yields better strategic and tactical decisions.
Big data with deep analytics is a journey that helps organizations solve key business issues and opportunities by converting data into insights to influence business actions and drive critical business outcomes. As organizations try to take advantage of the big data opportunity, they need not be overwhelmed by the various challenges that might await them.
Managers will need to start their journey by :
Identifying clear business need and value. Almost everything needs to be a business rather than a technology solution. Before companies start collecting big data, they should have a clear idea of what they want to do with it with from a business sense. Here’s what you need to consider:
Turn over part or all of big data solution delivery to business leaders. Project management and ownership from business (not IT) in big data solutions is the key for success. In the meantime, make sure to have clear alignment between business and IT.
Partner with business peers to identify opportunities and solutions. If we talk about big data, the impact of these projects should also be “big.” Create a cross-organization team and involve all stakeholders early in the game.
Value co-creation of value with customers. Overall business objective should always be about customers. If one of the initiatives is about big marketing outcome, than it should be about how to set up customer-centric marketing, how to provide targeted dynamic advertisement, how to engage customers and how to manage personalized shopping.
Start small – with an eye to scale quickly. While big data solutions may be quite advanced, everything else surrounding it – best practices, methodologies, org structures, etc. – is nascent. No one has all the answers, at least not yet. Understand why traditional business intelligence and data warehousing projects can’t solve a problem.
Small, simple and scalable. When launching big data initiatives, avoid 1) getting too complicated too fast, and 2) not being prepared to scale once a solution catches on. Big data solutions can quickly grow out of control since discovering value from data prompts wanting more data.
Identify what part of the business would benefit from quick wins. Look for opportunities that will show quick wins within no more than three months. Success brings more people to the table.
This is not a one-time implementation. Understand that this is a living and evolving organism that will grow exponentially very fast. It is a culture change in the company with the way that you collect and use data, and the way you make outcome-based decisions.
Develop a minimal set of big data governance directives upfront. Big data governance is a chicken-and-egg problem – you can’t govern or secure what you haven’t explored. However, exploring vast data sets without governance and security introduces risk.
Table 1: Common mistakes for analytics projects.
|Failing to build the need for big data within the organization|
|Islands of analytics with “Excel culture”|
|Data quality and reliability related issues|
|Not enough investigation on vendor products and rather than blindly taking the path of least resistance|
|Departmental thinking rather than looking at the big picture|
|Considering this as a one-time implementation rather than a living eco-system|
|Developing silo dashboards to answer a few questions rather than strategic, tactical and operational dashboards|
|Not establishing company ontology and definitions for “single version of truth” culture|
|Lack of vision and not having a strategy; not having a clear organizational communications plan|
|Lack of upfront planning; overlooking the development of governance and program oversight|
|Failure to re-organize for big data|
|Not establishing a formal training program|
|Ignoring the need to sell success and market the big data program|
|Not having the adequate architecture for data integration|
|Forgetting rapidly increasing complexities with …volume, velocity, variety, veracity, and many more|
New processes to manage open source risks. Most big data solutions are being built on open source software, but open source has both legal and skill implications as firms are: 1) exposed to risk due to intellectual property issues and complex licensing agreements; 2) concerned about liability if systems built on open source fail; and 3) required to use technology that is often early release and not enterprise-class.
New agile processes for solution delivery. Successful firms will embrace agile practices that allow end users of big data solutions to provide highly interactive inputs throughout the implementation process.
Integrate structured and unstructured data from multiple sources. Integration of data is one of the most important and also complex processes to serve efficient and effective decision-making. In terms of data, it includes machine data, sensor data, videos, audio, documents, enterprise content in call centers, e-mail messages, wikis and, indeed, larger volumes of transactional and application data.
Data sharing is key. In order for a company to build a big data ecosystem that drives business action, organizations have to share data.
Build a strong data infrastructure to host and manage data. Make sure to have secured and reliable in-house and/or hosted data (e.g., cloud) and information management infrastructure.
Think about what information do I collect today … and what analytics should I perform that can benefit me and others.
New security and compliance procedures to protect extreme-scale data. In order to succeed with big data, new processes must be developed that recognize and protect the special nature of extreme-scale data that may be largely unexplored.
Be ready to support rapid growth. Big data solutions can grow fast and exponentially. They can start as a pilot with a few terabytes of data, then becomes a petabyte very quickly. Since the same data can be used different ways and reanalyzed for new insights easily, nothing ever gets deleted.
Funding must move out of IT for big data success. Funding for these projects should come from outside of the CIO organization and move to a marketing or sales organization, for instance, so that the business has a vested stake in the game.
Create a road map that gradually builds the skills of your organization. It’s important to create a road map that allows you to gradually build the required skills within your staff, minimize risk and capitalize on previous successes to gain more support. In the organization, there will be new roles and responsibilities such as the data scientist, who possesses a blend of skills that includes statistics, applied mathematics and computer science.
This is different than any current decision support solution. With big data, organizations should look for new capabilities, such as: using advanced analytics to uncover patterns previously hidden; visualization and exploration to help the business find more complete answers, with new types and greater volumes of data to best represent the data to the user and highlight important patterns to the human eye; enable operational decision-making with on-demand stream data by making floor employees into analytic consumers; and turn insight into action to drive a decision – either with a manual step or an automated process. And most important be ready for rapidly increasing benefits and complexities from the six Vs.
What is Next in the Data Economy?
Organizations have access to a wealth of information, but they can’t get value out of it because it is sitting in its most raw form or in a semi-structured or unstructured format . As a result, they don’t even know whether it’s worth keeping.
So where is deep analytics for deep learning headed in the next few years? The exciting news is that many organizations are already realizing the value of big data analytics today. Insight-driven, information-centric initiatives will be deployed where the ability to capitalize on the six Vs of information will create new opportunities for organizations to exploit. By combining and integrating deep analytics, local rules, scoring, optimization techniques and machine learning with cognitive science into business processes and systems, decision management helps deliver decisions that are consistently optimized and aligned with the organization’s desired outcomes.
Social analytics will ensure businesses know how, when and where to creatively engage with individual consumers and social communities to foster trusted, one-to-one relationships and better understand and manage the way their companies are perceived. Integrating demographic and transactional data with what can be learned about attitudes and opinions allows organizations to truly understand the motivations and intents of its constituents to better serve them at the right time and place.
Deep analytics will help organizations uncover previously hidden patterns, identify classifications, associations and segmentations, and make highly accurate predictions from structured and unstructured information. Organizations will use real-time analysis of current activity to anticipate what will happen and identify drivers of various business outcomes so they can address the issues and challenges before they occur. Many decisions will be done automatically by computers that also have deep-learning capabilities.
When you are in a process of starting a big data journey, consider this question: What should our big data with deep analytics roadmap look like to achieve our objectives?
Haluk Demirkan (firstname.lastname@example.org) is a professor of Service Innovation and Business Analytics, and the founder and executive director of Center for Information Based Management at the Milgard School of Business, University of Washington-Tacoma. He has a Ph.D. in information systems and operations management from the University of Florida. He is a longtime member of INFORMS.
Bulent Dal (email@example.com) is a co-founder and general manager of Obase Analytical Solutions, Istanbul, Turkey. His expertise is in scientific retail analytical solutions. He has a Ph.D. in computer sciences engineering from Istanbul University.
Part of this article is excerpted with permission of the publisher, HBR Turkey, from Demirkan, H. and Dal, B., “Big Data, Big Opportunities, Big Decisions,” Harvard Business Review Turkish Edition (published in Turkish), March 2014.
- Gartner, Inc., 2013, “Gartner Predicts Business Intelligence and Analytics Will Remain Top Focus for CIOs Through 2017,” Dec. 16, 2013, http://www.gartner.com/newsroom/id/2637615.
- Demirkan, H. and Dal, B., “Big Data, Big Opportunities, Big Decisions,” Harvard Business Review Turkish Edition (published in Turkish), March 2014, pp. 28-30.
- Davenport, T., 2013, “Analytics, 3.0,” Harvard Business Review, December.
- 83September/October 2013 By Paul Kent (LEFT TO RIGHT), Radhika Kulkarni and Udo Sglavo We’re in the era of big data, but what do we mean by that? In our view, big data is a relative, not absolute, term. It means that the organization’s need to handle, store and analyze data…
- 74November/December 2013 By Gary Cokins Allow me to take a contrarian view to the rapid interest in applying analytics. Have we seen this kind of story before? Is there a rush to judgment that analytics is the elusive cure-all magic potion or panacea that management has been seeking to achieve…
- 65July/August 2013 Four things to look for when building an analytics team. By Andrew Jennings Companies in every industry from retail to banking are leveraging big data to improve the customer experience and enhance their bottom lines. Big data – high volume, high velocity (real time) and high variety (structured…
- 56As one of the premier indicators of the flourishing self-service movement within the datasphere, semantically enriched smart data lakes provide a means for business users to access and analyze big data on an unprecedented, enterprise-wide scale. The enduring relevance of these platforms is almost entirely based on the value that…
- 52July/August 2014 The story of how IBM not only survived but thrived by realizing business value from big data. By (l-r) Brenda Dietrich, Emily Plachy and Maureen Norton This is the story of how an iconic company founded more than a century ago, and once deemed a “dinosaur” that would…