Share with your friends


Analytics Magazine

Managing Large Data Sets: Companies don’t need more customer data – they need the ‘right data’

Organizations don’t need a lot of data; they just need the “right data.”

Organizations don’t need a lot of data; they just need the “right data.”

Chirag ShivalkerBy Chirag Shivalker

Companies and enterprises love to be called “data-driven.” Their business intelligence teams don’t refrain from including it in their strategy. But do they really know the meaning of being a “data-driven organization”? Though organizations succeed in generating and collecting a humongous amount of customer data, having big databases does not serve as the key enabler in making a business data-driven and digital.

It’s a sad state that companies invest millions of dollars in building and managing large data sets, but Forrester research suggests that these organizations analyze only about 12 percent of this data, whereas the rest of the 88 percent of data lives in silos of systems and departments. Lack of data management and analytics expertise, analytic solutions and “brutal” data silos are the reasons companies ignore a vast majority of their own data. Most often it’s a challenge for them to know what customer information is valuable and is best left ignored, and even if they somehow manage the data, they face various other challenges in using the data to their advantage.

Barriers to put data to work include:

  • complex business processes and IT landscape
  • customer data is spread over several departments, business processes and systems
  • insufficiently integrated customer-related systems and processes
  • lack of clear ownership and governance of customer data
  • lack of clarity with regard to data model for various data types
  • attributes to customer data are not defined appropriately
  • conventional systems do not support storage of enriched customer data
  • traditional systems do not have functionalities to really “master” the customer data
  • inflexible and difficult to adapt systems of data collection

The results of the above include:

  • duplicate customer records
  • inaccurate, incomplete, inconsistent and outdated customer data
  • outdated customer records across business applications
  • low-quality customer profiles
  • missing relationships with other data types
  • low level of automation of customer-related processes

Focus on Getting the ‘Right Data’

Organizations don’t need a lot of data; they just need the “right data.” A recent article in the Harvard Business Review opined that meaningful trusted data is more important than having a lot of data. The article quotes the example of Uber; the success of this taxi service is credited to having a lot of data and its capabilities to process huge data on a real-time basis. The surprising element, however, is that lots of data and processing is really not the key to Uber’s phenomenal growth. Analysis of the data of the taxi market concluded that a very small amount of the “right data” – dispatching cars – was required to run the show.

What separated Uber from other taxi companies was that the latter used complex ecosystems where continuous contact with drivers was mandatory. Uber identified this loophole and fixed it by asking for the right data in the form of: Who needs a ride? Where are they? Who is the Uber driver that is nearby?

In the case of retailers or e-commerce giants, data management experts can help them do this by cleansing, classifying, structuring, validating and integrating enterprise data generated from conventional resources, legacy resources and social media, which is imperative for organizations to make smarter, faster and stronger decisions. Business intelligence solutions facilitate quantitative analysis through predictive analytics, predictive modeling, business process modeling and statistical analysis.

One more reason why organizations should focus more on the “right data” is that collection and correction, if required, of data fields is always time-consuming and very costly. As an example, one of our clients, who has invested in more than 250 growth-stage software, Internet and data services companies, faced the challenge of reaching out to the target audience with the right data. Its investment teams were required to meet thousands of companies each year to select a handful to invest in. We conducted web research and applied analytics to the data collected and de-duped it to help the client with market insights and exposure to potential acquisitions and acquirers.

Customer Experience Management

The retail industry has entered the “age of the customer.” Customer eXperience (CX), clubbed with the identification of patterns of researching, buying and using products or services – what we call the customer journey – has become more than important. On one hand, customers are steering the retail economy, and on the other, they also use increased number of channels to communicate and interact. This has increased the relevance of customer data manifold.

Delivering optimum customer services is the key aspect on which companies are competing today. Gartner’s research suggests that nearly 90 percent of companies are of the opinion that CX will grow to be the prime domain for competition, and rich customer data and analytic insights through data visualization and dashboards would be the main weapon to win this CX battle. It will also help organizations in recognizing the customers and managing relationships with them, differentiating and offering the right products and services and assist in treating every customer as a unique individual.

It would not be wrong to say that CX initiatives can succeed only with the help of a 360-degree view of the customer. It is also because it empowers a wide plethora of departments and functions to identify customers and to consistently interact with them. It is interesting to know that the progress of interactions across sales, customer service, logistics and several other functions make the customer feel reassured that the organization they are dealing with “knows” them. CX initiatives improve customer satisfaction scores, NET promoter scores, customer retention rates and the number of reference customers.

Start with ‘Right Customer Data’

Marketers firmly believe that focusing on the customer journey is very important, but great customer interactions start with great customer data. Collecting trustful data, processing that data efficiently and then maintaining that data to get actionable insights are the key challenges to identify, segment, personalize and target customers and prospects. Retailers can leverage analytic solutions facilitating data modeling, eCommerce dashboards, market dashboards and visual dashboards to spend their marketing budget in a more efficient way, while improving sales and loyalty – all at the same time. y

Chirag Shivalker, digital content head at Hi-Tech BPO with more than 18 years of experience, writes about agile methodologies of developing enterprise-wide data management solutions including robust analytics and reporting environments, and integrated analytics solutions.

Analytics data science news articles

Related Posts

  • 84
    With the rise of big data – and the processes and tools related to utilizing and managing large data sets – organizations are recognizing the value of data as a critical business asset to identify trends, patterns and preferences to drive improved customer experiences and competitive advantage. The problem is,…
    Tags: data
  • 84
    Features   Customer data: volume vs. value By Chirag Shivalker Companies invest millions of dollars in building and managing large data sets, but they don’t need more customer data; what they really need is the “right data.” Big data: products, services, operations By Dave Oswill What data scientists and engineers…
    Tags: data, customer
  • 82
    The Internet of Things (IoT) is considered to be the next revolution that touches every part of our daily life, from restocking ice cream to warning of pollutants. Analytics professionals understand the importance of data, especially in a complicated field such as healthcare. This article offers a framework on integrating…
    Tags: data
  • 79
    Businesses are greatly expanding the autonomous capabilities of their products, services and manufacturing processes to better optimize their reliability and efficiency. The processing of big data is playing an integral role in developing these prescriptive analytics. As a result, data scientists and engineers should pay attention to the following aspects…
    Tags: data
  • 77
    Frontline Systems releases Analytic Solver V2018 for Excel Frontline Systems, developer of the Solver in Microsoft Excel, recently released Analytic Solver V2018, its full product line of predictive and prescriptive analytics tools that work in Microsoft Excel. The new release includes a visual editor for multi-stage “data science workflows” (also…
    Tags: data


Finalists for 2018 Syngenta Crop Challenge announced

Syngenta and the Analytics Society of INFORMS this week announced the finalists for the 2018 Syngenta Crop Challenge in Analytics. Now in its third year, the competition aims to address the challenge of achieving global food security by fostering cross-industry collaboration between agriculture and advanced analytics experts. Read more →

International Conference to spotlight O.R., analytics and AI

The 2018 INFORMS International Conference will be held in the world-class International Convention Center (TICC) and the Grand Hyatt Taipei in beautiful Taipei City, Taiwan, on June 17-20. The conference theme is “A Better World Through Operations Research, Analytics and Artificial Intelligence.” The conference offers excellent opportunities to learn about the emerging O.R. and A.I. technologies and applications from more than 40 invited, sponsored and contributed clusters featuring more than 600 talks. Read more →

How math education can catch up to the 21st century

Big data analytics is increasingly becoming a trending practice that society is adopting, and Daniel Kunin, a scholar at Stanford University and the creator of the online platform, Seeing Theory, is using creative and innovative ways to teach statistics and probability relevant to a changing world. In a recent interview with CMRubinWorld, Kunin discusses math education today and how he believes it can be improved in order to both foster curiosity and be more relevant now and going forward. Read more →



2018 INFORMS Conference on Business Analytics and Operations Research
April 15-17, 2018, Baltimore


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to