Share with your friends


Analytics Magazine

Supply Chain Management: New math, big data

July/August 2011


Across the supply chain, manufacturers and retailers are exploiting the vital raw material of the information economy.

Robert F. ByrneBy Robert F. Byrne

Manufacturer and retailer data from across the supply chain is a powerful source of information, providing a picture of product movement in near real-time. Access to the data is just the first step, because the volume of data available each day is overwhelming and much of it is noisy. Manufacturers need a structured and touchless approach to analyze masses of data every day, cut through the noise, and transform it to meaningful information that can be used to create an actionable response based on that information.

The Era of Big Data

We are living through a data explosion. According to a recent New York Times article, “New Ways to Exploit Raw Data May Bring Surge of Innovation,” “Data is a vital raw material of the information economy, much as coal and iron ore were in the Industrial Revolution. But the business world is just beginning to learn how to process it all. The current data surge is coming from sophisticated computer tracking of shipments, sales, suppliers and customers, as well as email, Web traffic and social network comments. The quantity of data doubles every 1.2 years, by one estimate.”

To put this in perspective, the McKinsey Global Institute estimates that new data stored by enterprises exceeded 7 exabytes of data globally in 2010, with consumers adding another 6 exabytes. An exabyte is a unit of information or computer storage equal to one quintillion bytes (one billion gigabytes, one million terabytes). The amount of new data stored in 2010 would fill more than 60,000 U.S. Libraries of Congress.

The manufacturing sector and the supply chain in particular are areas that can benefit from big data. The amount and types of data that can be used to manage the supply chain have mushroomed over the past years and grow daily. A Wal-Mart supercenter can carry 100,000 SKUs and Wal-Mart’s warehouses include some 2.5 petabytes of information. Tesco generates more than 1.5 billion new items of data every month. Supply chain specific data such as orders, shipments and inventory levels from both manufacturer and retailers provide a rich set of demand signals about products across the value chain — from raw materials to consumer purchases.

External sources such as weather systems can be tracked and the supply chain can respond to avoid hazardous conditions, areas where stores are closed or areas where demand spikes for products made for use in inclement weather. Knowledge of outbreaks of flu can be used to determine which areas may need more chicken soup or cough drops, while details about road construction and bridge closings can be used to reroute shipments so that products continue to reach store shelves on time. The emergence of social media offers yet a new source of potential demand signals.

There is no lack of data in the supply chain — what’s missing are the mathematics and structured systems to convert the masses of raw data into meaningful and actionable information that suppliers, manufacturers and retailers can use to make critical business decisions about production, replenishment, distribution and inventory holdings.

Forecast Accuracy Remains Problematic

Accurate forecasting is at the heart of all business decisions but remains the Achilles heel of the supply chain. Despite the existence of abundant information about product movement, average forecast error remains 48 percent for consumer products companies. Worse yet, for 85 percent of their SKUs, forecast error is greater than 60 percent. Getting forecasts right is a serious business — these companies use their demand forecast to determine which products to make, how much to produce, when to make them and where to ship them. Inaccurate forecasts wreak havoc in the supply chain. Over-forecasts result in excess inventory being produced and held, tying up cash and lowering return on assets. Under-forecasts lead to unscheduled changeovers to meet unexpected demand — increasing costs and disrupting production. Worse yet, stock-outs result in lost sales, damaged market share and risk jeopardizing collaboration efforts with supply chain partners.

According to the IHL Group, “Consumers loathe out-of-stocks: in survey after survey, consumers say that not being able to purchase a product that a store is expected to have, or has been promoted in advertisements, is second only to standing in long lines on their list of shopping frustrations.” Frustrated consumers may choose another brand or shop at a different store, costing both manufacturers and retailers revenue. Frustrated retailers may offer better shelf space or more promotional opportunities to manufacturers that do not run out of stock, penalizing the manufacturer for the inaccurate forecast. Yet despite these consequences, BMO Capital reports that out-of-stocks remain greater than 8 percent and this number has not changed in the last 15 years.

Traditional time series forecasting analyzes historical patterns to forecast demand, adjusted by planners for promotions and expert input from sales. But the fundamental assumption that history can predict the future is inaccurate. Natural disasters — hurricanes, volcanoes, flooding — and unexpected competitive activities demonstrate the need for agile planning processes that respond to what is happening today. These events happen in the present and need to be accounted for to create a clear picture of product demand.

Supply Chain ManagementCreating an Actionable Response Using Big Data

Maximizing the value of big data requires integrating data from internal and external sources, accelerating the frequency of planning cycles and investing in new technologies and techniques to analyze data automatically. According to a report by McKinsey Global Institute (“Big Data: The next frontier for innovation, competition, and productivity,” May 2011), “By taking into account data from across the value chain (potentially through collaborative supply chain management and planning), manufacturers can smooth spiky order patterns. The benefits of doing so will ripple through the value chain, helping manufacturers to use cash more effectively and to deliver a higher level of service. Best-in-class manufacturers are also accelerating the frequency of planning cycles to synchronize them with production cycles. Indeed, some manufacturers are using near-real-time data to adjust production.”

Large manufacturers have begun analyzing data from throughout the supply chain, but initially invested primarily in business intelligence applications to mine the data in support of specific activities such as confirming whether promotional lift was on target for a specific product or to help identify phantom inventory. While helpful, these applications often require skilled analytics personnel to effectively use the systems and, more importantly, provide value on an ad hoc basis. In short, these techniques have limited scalability and would require an army of analysts to sift through the sheer volume of data to determine what data affects actual demand at each product-location.

While rules of thumb predictions can be helpful, the factors affecting demand are different for every product and can vary by location. Demand signals with the strongest influence on predictability are sometimes counterintuitive and can often be factors that a human analyst would not consider relevant or would guess incorrectly. POS data is almost always valuable, but its influence as a predictor varies considerably for different products, locations or even retailer channels. For example, POS may be a good predictor for lipstick which comes in hundreds of shades, whereas fast-moving products such as paper towels might respond better to warehouse withdrawals. Seasonal products like turkey or whipped cream behave differently again. Further, the influence of specific demand signals varies depending on the forecast horizon. The key is structured mathematics designed to process masses of data and solve the complex problem of determining which signals are predictors for each product, location and customer combination.

Structured mathematics overcomes another traditional issue for forecasts — algorithms have no ulterior motives and their feelings do not get hurt. Politics, preconceived notions and intuition invariably shape human perceptions and bring their own biases to the forecasting process. Free of these influences, math creates a forecast that relies on data alone. Further, the scale of the data is beyond the capacity of the human mind to process in a reasonable time frame. “In this world of big data, relational databases and desktop applications — spreadsheets, statistical packages and reporting – are insufficient,” says Lora Cecere, Partner, Altimeter Group. “Instead, it requires the use of parallel software running on tens, hundreds or even thousands of servers. It is the world of terabytes, exabytes and zettabytes of data.”

The justification for the significant investment in capturing daily demand signals comes from using this data to respond quickly and efficiently to changes in demand. Cecere explains that market leaders who invested in the ability to sense demand, sensed market changes five times faster and corrected their supply chains three to 12 months sooner than followers and laggards during the recession. These leaders responded to the economic downturn a full year ahead of some of their competitors, adjusting their product mix and gaining a competitive advantage.

Realizing this potential requires scalable applications designed to use daily-data-daily. Access to daily demand signals would not have helped these manufacturers get ahead of their competition if it required months or even weeks to analyze the data. Creating and publishing daily forecasts requires automation; it is a prerequisite for the structured use of big data. For large manufacturers with many SKUs, it is simply impossible to review forecasts for every product and location on a daily basis.

Benefits of Using Big Data in Forecasting

In today’s volatile environment, companies are faced with the risks of rising interest rates, commodity prices and demand uncertainty. Interest rates are clearly outside a company’s control. While hedging can provide protection for commodity prices, its effects are only short-term. The pursuit of accurate forecasts to mitigate demand uncertainty remains one of the last major levers at their disposal. Though forecasts have been traditionally seen as “always wrong,” the use of big data and demand sensing’s better math provides a step change in forecast accuracy and is rewriting the rules. Granted, these new forecasts are still wrong, but they are a lot less wrong.

The benefits are tangible and financially significant. By using big data to sense demand, companies reduce forecast error by 40 percent or more. The most obvious way companies monetize this improvement is to reduce inventory while maintaining or improving service levels. This strengthens the balance sheet by freeing up cash flow and improving return on invested capital and boosts the income statement by eliminating production and carrying costs of inventory whose sole purpose is to mitigate demand uncertainty. Having the product in the right place the first time also avoids expensive transshipments and expedites or unplanned manufacturing changeovers.

Further, controlling demand volatility through better math might just be the single largest contribution a company can make to meet its corporate sustainability goals. Piles of inventory are more than just piles of capital; they are piles of carbon and water. For multinational companies with large and complex supply chains, the scale of carbon and water tied up in inventory is enormous. Reducing inventory through better forecasts results in a significant, one-time reduction in the company’s environmental footprint, turning the core supply chain into a sustainability asset for a company’s total product portfolio, not just the “green” items.


The data explosion in the supply chain has provided companies with an unprecedented opportunity to control their demand volatility. Accurate forecasts are finally available to those willing to invest in technology that converts this data into actionable information — leading to improved efficiencies, financial performance and sustainable operations. Never before have manufacturers had access to so much information about product demand and the external forces that can impact demand — from the manufacturer’s own supply chain, retailer and distributor supply chains. Welcome to the age of big data. With the right math, it is a game changer.

Robert F. Byrne is president and CEO of Terra Technology (, a provider of supply chain solutions for consumer products companies.


Related Posts

  • 48
    International Data Corporation (IDC) recently released a worldwide Big Data technology and services forecast showing the market is expected to grow from $3.2 billion in 2010 to $16.9 billion in 2015. This represents a compound annual growth rate (CAGR) of 40 percent or about seven times that of the overall…
    Tags: data, big
  • 46
    Many organizations have noticed that the data they own and how they use it can make them different than others to innovate, to compete better and to stay in business. That’s why organizations try to collect and process as much data as possible, transform it into meaningful information with data-driven…
    Tags: data, big
  • 43
    New research from Accenture Interactive indicates that offering personalized experiences will translate to higher sales if businesses remove unintended barriers consumers report. While most consumers welcome personalized offerings, many complain of having received irrelevant recommendations and experienced an overwhelming choice of options when visiting brands and retailers’ websites.
    Tags: data, retailers, retailer
  • 42
    September/October 2012 How to make choices and investments that deliver on expectations. By Sally Taylor-Shoff (left) and Shalini Raghavan The most successful retailers today are increasing response rates to their offers and driving profitability by using Big Data and predictive analytics to make relevant, personalized, and precisely timed offers to…
    Tags: retailers, retailer, data, big
  • 40
    The Panama Papers, the unprecedented leak of 11.5 million files from the database of the global law firm Mossack Fonseca, opened up the offshore tax accounts of the rich, famous and powerful – laying bare how they have exploited secretive offshore tax regimes for decades.
    Tags: data, big


Using machine learning and optimization to improve refugee integration

Andrew C. Trapp, a professor at the Foisie Business School at Worcester Polytechnic Institute (WPI), received a $320,000 National Science Foundation (NSF) grant to develop a computational tool to help humanitarian aid organizations significantly improve refugees’ chances of successfully resettling and integrating into a new country. Built upon ongoing work with an international team of computer scientists and economists, the tool integrates machine learning and optimization algorithms, along with complex computation of data, to match refugees to communities where they will find appropriate resources, including employment opportunities. Read more →

Gartner releases Healthcare Supply Chain Top 25 rankings

Gartner, Inc. has released its 10th annual Healthcare Supply Chain Top 25 ranking. The rankings recognize organizations across the healthcare value chain that demonstrate leadership in improving human life at sustainable costs. “Healthcare supply chains today face a multitude of challenges: increasing cost pressures and patient expectations, as well as the need to keep up with rapid technology advancement, to name just a few,” says Stephen Meyer, senior director at Gartner. Read more →

Meet CIMON, the first AI-powered astronaut assistant

CIMON, the world’s first artificial intelligence-enabled astronaut assistant, made its debut aboard the International Space Station. The ISS’s newest crew member, developed and built in Germany, was called into action on Nov. 15 with the command, “Wake up, CIMON!,” by German ESA astronaut Alexander Gerst, who has been living and working on the ISS since June 8. Read more →



INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.


Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)


CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:

For more information, go to