Share with your friends


Analytics Magazine

Managing Large Data Sets: Companies don’t need more customer data – they need the ‘right data’

Organizations don’t need a lot of data; they just need the “right data.”

Organizations don’t need a lot of data; they just need the “right data.”

Chirag ShivalkerBy Chirag Shivalker

Companies and enterprises love to be called “data-driven.” Their business intelligence teams don’t refrain from including it in their strategy. But do they really know the meaning of being a “data-driven organization”? Though organizations succeed in generating and collecting a humongous amount of customer data, having big databases does not serve as the key enabler in making a business data-driven and digital.

It’s a sad state that companies invest millions of dollars in building and managing large data sets, but Forrester research suggests that these organizations analyze only about 12 percent of this data, whereas the rest of the 88 percent of data lives in silos of systems and departments. Lack of data management and analytics expertise, analytic solutions and “brutal” data silos are the reasons companies ignore a vast majority of their own data. Most often it’s a challenge for them to know what customer information is valuable and is best left ignored, and even if they somehow manage the data, they face various other challenges in using the data to their advantage.

Barriers to put data to work include:

  • complex business processes and IT landscape
  • customer data is spread over several departments, business processes and systems
  • insufficiently integrated customer-related systems and processes
  • lack of clear ownership and governance of customer data
  • lack of clarity with regard to data model for various data types
  • attributes to customer data are not defined appropriately
  • conventional systems do not support storage of enriched customer data
  • traditional systems do not have functionalities to really “master” the customer data
  • inflexible and difficult to adapt systems of data collection

The results of the above include:

  • duplicate customer records
  • inaccurate, incomplete, inconsistent and outdated customer data
  • outdated customer records across business applications
  • low-quality customer profiles
  • missing relationships with other data types
  • low level of automation of customer-related processes

Focus on Getting the ‘Right Data’

Organizations don’t need a lot of data; they just need the “right data.” A recent article in the Harvard Business Review opined that meaningful trusted data is more important than having a lot of data. The article quotes the example of Uber; the success of this taxi service is credited to having a lot of data and its capabilities to process huge data on a real-time basis. The surprising element, however, is that lots of data and processing is really not the key to Uber’s phenomenal growth. Analysis of the data of the taxi market concluded that a very small amount of the “right data” – dispatching cars – was required to run the show.

What separated Uber from other taxi companies was that the latter used complex ecosystems where continuous contact with drivers was mandatory. Uber identified this loophole and fixed it by asking for the right data in the form of: Who needs a ride? Where are they? Who is the Uber driver that is nearby?

In the case of retailers or e-commerce giants, data management experts can help them do this by cleansing, classifying, structuring, validating and integrating enterprise data generated from conventional resources, legacy resources and social media, which is imperative for organizations to make smarter, faster and stronger decisions. Business intelligence solutions facilitate quantitative analysis through predictive analytics, predictive modeling, business process modeling and statistical analysis.

One more reason why organizations should focus more on the “right data” is that collection and correction, if required, of data fields is always time-consuming and very costly. As an example, one of our clients, who has invested in more than 250 growth-stage software, Internet and data services companies, faced the challenge of reaching out to the target audience with the right data. Its investment teams were required to meet thousands of companies each year to select a handful to invest in. We conducted web research and applied analytics to the data collected and de-duped it to help the client with market insights and exposure to potential acquisitions and acquirers.

Customer Experience Management

The retail industry has entered the “age of the customer.” Customer eXperience (CX), clubbed with the identification of patterns of researching, buying and using products or services – what we call the customer journey – has become more than important. On one hand, customers are steering the retail economy, and on the other, they also use increased number of channels to communicate and interact. This has increased the relevance of customer data manifold.

Delivering optimum customer services is the key aspect on which companies are competing today. Gartner’s research suggests that nearly 90 percent of companies are of the opinion that CX will grow to be the prime domain for competition, and rich customer data and analytic insights through data visualization and dashboards would be the main weapon to win this CX battle. It will also help organizations in recognizing the customers and managing relationships with them, differentiating and offering the right products and services and assist in treating every customer as a unique individual.

It would not be wrong to say that CX initiatives can succeed only with the help of a 360-degree view of the customer. It is also because it empowers a wide plethora of departments and functions to identify customers and to consistently interact with them. It is interesting to know that the progress of interactions across sales, customer service, logistics and several other functions make the customer feel reassured that the organization they are dealing with “knows” them. CX initiatives improve customer satisfaction scores, NET promoter scores, customer retention rates and the number of reference customers.

Start with ‘Right Customer Data’

Marketers firmly believe that focusing on the customer journey is very important, but great customer interactions start with great customer data. Collecting trustful data, processing that data efficiently and then maintaining that data to get actionable insights are the key challenges to identify, segment, personalize and target customers and prospects. Retailers can leverage analytic solutions facilitating data modeling, eCommerce dashboards, market dashboards and visual dashboards to spend their marketing budget in a more efficient way, while improving sales and loyalty – all at the same time. y

Chirag Shivalker, digital content head at Hi-Tech BPO with more than 18 years of experience, writes about agile methodologies of developing enterprise-wide data management solutions including robust analytics and reporting environments, and integrated analytics solutions.

Analytics data science news articles

Related Posts

  • 84
    With the rise of big data – and the processes and tools related to utilizing and managing large data sets – organizations are recognizing the value of data as a critical business asset to identify trends, patterns and preferences to drive improved customer experiences and competitive advantage. The problem is,…
    Tags: data
  • 84
    Features Customer data: volume vs. value By Chirag Shivalker Companies invest millions of dollars in building and managing large data sets, but they don’t need more customer data; what they really need is the “right data.” Big data: products, services, operations By Dave Oswill What data scientists and engineers need…
    Tags: data, customer
  • 82
    The Internet of Things (IoT) is considered to be the next revolution that touches every part of our daily life, from restocking ice cream to warning of pollutants. Analytics professionals understand the importance of data, especially in a complicated field such as healthcare. This article offers a framework on integrating…
    Tags: data
  • 79
    Businesses are greatly expanding the autonomous capabilities of their products, services and manufacturing processes to better optimize their reliability and efficiency. The processing of big data is playing an integral role in developing these prescriptive analytics. As a result, data scientists and engineers should pay attention to the following aspects…
    Tags: data
  • 77
    Frontline Systems releases Analytic Solver V2018 for Excel Frontline Systems, developer of the Solver in Microsoft Excel, recently released Analytic Solver V2018, its full product line of predictive and prescriptive analytics tools that work in Microsoft Excel. The new release includes a visual editor for multi-stage “data science workflows” (also…
    Tags: data


Fighting terrorists online: Identifying extremists before they post content

New research has found a way to identify extremists, such as those associated with the terrorist group ISIS, by monitoring their social media accounts, and can identify them even before they post threatening content. The research, “Finding Extremists in Online Social Networks,” which was recently published in the INFORMS journal Operations Research, was conducted by Tauhid Zaman of the MIT, Lt. Col. Christopher E. Marks of the U.S. Army and Jytte Klausen of Brandeis University. Read more →

Syrian conflict yields model for attrition dynamics in multilateral war

Based on their study of the Syrian Civil War that’s been raging since 2011, three researchers created a predictive model for multilateral war called the Lanchester multiduel. Unless there is a player so strong it can guarantee a win regardless of what others do, the likely outcome of multilateral war is a gradual stalemate that culminates in the mutual annihilation of all players, according to the model. Read more →

SAS, Samford University team up to generate sports analytics talent

Sports teams try to squeeze out every last bit of talent to gain a competitive advantage on the field. That’s also true in college athletic departments and professional team offices, where entire departments devoted to analyzing data hunt for sports analytics experts that can give them an edge in a game, in the stands and beyond. To create this talent, analytics company SAS will collaborate with the Samford University Center for Sports Analytics to support teaching, learning and research in all areas where analytics affects sports, including fan engagement, sponsorship, player tracking, sports medicine, sports media and operations. Read more →



INFORMS Annual Meeting
Nov. 4-7, 2018, Phoenix

Winter Simulation Conference
Dec. 9-12, 2018, Gothenburg, Sweden


Making Data Science Pay
Oct. 29 -30, 12 p.m.-5 p.m.

Applied AI & Machine Learning | Comprehensive
Starts Oct. 29, 2018 (live online)

The Analytics Clinic
Citizen Data Scientists | Why Not DIY AI?
Nov. 8, 2018, 11 a.m. – 12:30 p.m.

Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to