Share with your friends


Analytics Magazine

Managing Large Data Sets: Companies don’t need more customer data – they need the ‘right data’

Organizations don’t need a lot of data; they just need the “right data.”

Organizations don’t need a lot of data; they just need the “right data.”

Chirag ShivalkerBy Chirag Shivalker

Companies and enterprises love to be called “data-driven.” Their business intelligence teams don’t refrain from including it in their strategy. But do they really know the meaning of being a “data-driven organization”? Though organizations succeed in generating and collecting a humongous amount of customer data, having big databases does not serve as the key enabler in making a business data-driven and digital.

It’s a sad state that companies invest millions of dollars in building and managing large data sets, but Forrester research suggests that these organizations analyze only about 12 percent of this data, whereas the rest of the 88 percent of data lives in silos of systems and departments. Lack of data management and analytics expertise, analytic solutions and “brutal” data silos are the reasons companies ignore a vast majority of their own data. Most often it’s a challenge for them to know what customer information is valuable and is best left ignored, and even if they somehow manage the data, they face various other challenges in using the data to their advantage.

Barriers to put data to work include:

  • complex business processes and IT landscape
  • customer data is spread over several departments, business processes and systems
  • insufficiently integrated customer-related systems and processes
  • lack of clear ownership and governance of customer data
  • lack of clarity with regard to data model for various data types
  • attributes to customer data are not defined appropriately
  • conventional systems do not support storage of enriched customer data
  • traditional systems do not have functionalities to really “master” the customer data
  • inflexible and difficult to adapt systems of data collection

The results of the above include:

  • duplicate customer records
  • inaccurate, incomplete, inconsistent and outdated customer data
  • outdated customer records across business applications
  • low-quality customer profiles
  • missing relationships with other data types
  • low level of automation of customer-related processes

Focus on Getting the ‘Right Data’

Organizations don’t need a lot of data; they just need the “right data.” A recent article in the Harvard Business Review opined that meaningful trusted data is more important than having a lot of data. The article quotes the example of Uber; the success of this taxi service is credited to having a lot of data and its capabilities to process huge data on a real-time basis. The surprising element, however, is that lots of data and processing is really not the key to Uber’s phenomenal growth. Analysis of the data of the taxi market concluded that a very small amount of the “right data” – dispatching cars – was required to run the show.

What separated Uber from other taxi companies was that the latter used complex ecosystems where continuous contact with drivers was mandatory. Uber identified this loophole and fixed it by asking for the right data in the form of: Who needs a ride? Where are they? Who is the Uber driver that is nearby?

In the case of retailers or e-commerce giants, data management experts can help them do this by cleansing, classifying, structuring, validating and integrating enterprise data generated from conventional resources, legacy resources and social media, which is imperative for organizations to make smarter, faster and stronger decisions. Business intelligence solutions facilitate quantitative analysis through predictive analytics, predictive modeling, business process modeling and statistical analysis.

One more reason why organizations should focus more on the “right data” is that collection and correction, if required, of data fields is always time-consuming and very costly. As an example, one of our clients, who has invested in more than 250 growth-stage software, Internet and data services companies, faced the challenge of reaching out to the target audience with the right data. Its investment teams were required to meet thousands of companies each year to select a handful to invest in. We conducted web research and applied analytics to the data collected and de-duped it to help the client with market insights and exposure to potential acquisitions and acquirers.

Customer Experience Management

The retail industry has entered the “age of the customer.” Customer eXperience (CX), clubbed with the identification of patterns of researching, buying and using products or services – what we call the customer journey – has become more than important. On one hand, customers are steering the retail economy, and on the other, they also use increased number of channels to communicate and interact. This has increased the relevance of customer data manifold.

Delivering optimum customer services is the key aspect on which companies are competing today. Gartner’s research suggests that nearly 90 percent of companies are of the opinion that CX will grow to be the prime domain for competition, and rich customer data and analytic insights through data visualization and dashboards would be the main weapon to win this CX battle. It will also help organizations in recognizing the customers and managing relationships with them, differentiating and offering the right products and services and assist in treating every customer as a unique individual.

It would not be wrong to say that CX initiatives can succeed only with the help of a 360-degree view of the customer. It is also because it empowers a wide plethora of departments and functions to identify customers and to consistently interact with them. It is interesting to know that the progress of interactions across sales, customer service, logistics and several other functions make the customer feel reassured that the organization they are dealing with “knows” them. CX initiatives improve customer satisfaction scores, NET promoter scores, customer retention rates and the number of reference customers.

Start with ‘Right Customer Data’

Marketers firmly believe that focusing on the customer journey is very important, but great customer interactions start with great customer data. Collecting trustful data, processing that data efficiently and then maintaining that data to get actionable insights are the key challenges to identify, segment, personalize and target customers and prospects. Retailers can leverage analytic solutions facilitating data modeling, eCommerce dashboards, market dashboards and visual dashboards to spend their marketing budget in a more efficient way, while improving sales and loyalty – all at the same time. y

Chirag Shivalker, digital content head at Hi-Tech BPO with more than 18 years of experience, writes about agile methodologies of developing enterprise-wide data management solutions including robust analytics and reporting environments, and integrated analytics solutions.

Analytics data science news articles

Related Posts

  • 84
    Features Customer data: volume vs. value By Chirag Shivalker Companies invest millions of dollars in building and managing large data sets, but they don’t need more customer data; what they really need is the “right data.” Big data: products, services, operations By Dave Oswill What data scientists and engineers need…
    Tags: data, customer
  • 84
    With the rise of big data – and the processes and tools related to utilizing and managing large data sets – organizations are recognizing the value of data as a critical business asset to identify trends, patterns and preferences to drive improved customer experiences and competitive advantage. The problem is,…
    Tags: data
  • 82
    The Internet of Things (IoT) is considered to be the next revolution that touches every part of our daily life, from restocking ice cream to warning of pollutants. Analytics professionals understand the importance of data, especially in a complicated field such as healthcare. This article offers a framework on integrating…
    Tags: data
  • 79
    Businesses are greatly expanding the autonomous capabilities of their products, services and manufacturing processes to better optimize their reliability and efficiency. The processing of big data is playing an integral role in developing these prescriptive analytics. As a result, data scientists and engineers should pay attention to the following aspects…
    Tags: data
  • 77
    Frontline Systems releases Analytic Solver V2018 for Excel Frontline Systems, developer of the Solver in Microsoft Excel, recently released Analytic Solver V2018, its full product line of predictive and prescriptive analytics tools that work in Microsoft Excel. The new release includes a visual editor for multi-stage “data science workflows” (also…
    Tags: data


Using machine learning and optimization to improve refugee integration

Andrew C. Trapp, a professor at the Foisie Business School at Worcester Polytechnic Institute (WPI), received a $320,000 National Science Foundation (NSF) grant to develop a computational tool to help humanitarian aid organizations significantly improve refugees’ chances of successfully resettling and integrating into a new country. Built upon ongoing work with an international team of computer scientists and economists, the tool integrates machine learning and optimization algorithms, along with complex computation of data, to match refugees to communities where they will find appropriate resources, including employment opportunities. Read more →

Gartner releases Healthcare Supply Chain Top 25 rankings

Gartner, Inc. has released its 10th annual Healthcare Supply Chain Top 25 ranking. The rankings recognize organizations across the healthcare value chain that demonstrate leadership in improving human life at sustainable costs. “Healthcare supply chains today face a multitude of challenges: increasing cost pressures and patient expectations, as well as the need to keep up with rapid technology advancement, to name just a few,” says Stephen Meyer, senior director at Gartner. Read more →

Meet CIMON, the first AI-powered astronaut assistant

CIMON, the world’s first artificial intelligence-enabled astronaut assistant, made its debut aboard the International Space Station. The ISS’s newest crew member, developed and built in Germany, was called into action on Nov. 15 with the command, “Wake up, CIMON!,” by German ESA astronaut Alexander Gerst, who has been living and working on the ISS since June 8. Read more →



INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.


Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to