Share with your friends










Submit

Analytics Magazine

Software Survey: Forecasting 2018

Survey reflects changes in the software market, but it also puts greater emphasis on the requirements of the practicing forecaster.

By Robert Fildes, Oliver Schaer and Ivan Svetunkov

A core principle of forecasting is to look at past history to understand and predict.

A core principle of forecasting is to look at past history to understand and predict.
Source. ThinkStock

Forecasting software has long been an essential part of high-end analytics and is now making a major contribution to the fast-developing area of business analytics. Forecasts are a necessary input into almost all planning decisions, be they long- or short-term, but they are also needed in performance analysis and revenue optimization. Forecasting methods should, therefore, form an important part of all university or practitioner courses in analytics. Suitable software is at the heart of the forecasting activity, whether the practitioner works in demand forecasting in the supply chain or as a business or government analyst.

Interestingly, the history of forecasting is to a certain extent written by those who have developed user-friendly software packages early in the dissemination of a new method. For example, the early development of Box-Jenkins ARIMA software ensured rapid adoption while the paucity of state space software limited its adoption, until the development of open source software in R (Hyndman and Khandakar, 2008 [3]). But forecasting software is not homogeneous, either in terms of the methods implemented but more importantly, in terms of the envisaged applications and it is a fast-developing field. For a number of years, OR/MS Today (the membership magazine of INFORMS) has been surveying forecasting software suppliers to offer an overview of the latest features and market trends (most recently Fry and Mehrotra, 2016 [1]). In this latest survey, INFORMS attempted to identify all forecasting software providers of packages that included time series forecasting, whether research-oriented software or software designed for operational organizational planning. Of course, only a proportion responded within the deadline. An updated list of vendor submissions is available online at: https://bit.ly/2Jpl66p.

A core principle of forecasting is to look at past history to understand and predict. The 2016 survey by Fry and Mehrotra included some aspects that have become redundant, for example, flexible data input and output. But this is now straightforward, and we have dropped these issues from the current questionnaire. Fry and Mehrotra also identified a number of trends in the market: “integration,” where the forecasts link into the planning system; “automation,” where the package includes automatic selection from a range of methods; “visualization” for an extended range of graphics; “virtualization,” where the vendor’s offering is delivered through the cloud, or in its more extreme version, the vendor virtually manages the forecasting process from beginning to end. We now see all of these features in current software.

The 2018 survey has been redesigned to reflect changes in the software market, but it also puts greater emphasis on the requirements of the practicing forecaster. Software for forecasting comes in a variety of forms – no single software product includes all the features that a user might want because different users have different requirements. In surveys of practitioners, Excel is usually named as the most commonly used forecasting package (Weller and Crone, 2012 [6]) although it is best thought of as a data handling and analysis program, rather than providing forecasting; its built-in forecasting routines remain limited, even in its latest 2016 incarnation. In addition, its statistical features have weaknesses (see McCullough and Heiser, 2008 [4]).

If Excel and its built-in routines are inadequate, forecasters can attempt to build their own methods using macros. This is not recommended because even simple methods, such as seasonal exponential smoothing, have their pitfalls. Nor is it straightforward to scale up to the number of time series needing forecasting for many applications. Instead, forecasting-specific packages should do the trick, but now the choice is extensive; it can usefully be broken down into a number of different categories that reflect both the specialization for which the package is designed and its focus of application. For example:

General purpose statistical package. These packages include a wide range of statistical techniques supplemented by forecasting routines. Sometimes there is a more specialized add-in (costing more) that enhances the basic system. Typically, the base general-purpose packages do not include the full range of forecasting methods and the additional features that a more advanced user might require. Market leaders include Minitab, SAS, SPSS, STATA and StatGraphics.

Specialized forecasting software. These may focus on a single technique or a range of methods. They include Autobox (which is an automatic Box-Jenkins ARIMA modeling and forecasting package), ForecastPro (which includes a range of methods), specialist neural network packages (such as Alyuda or Analytic Solver Data Mining) and state space modelling (in STAMP, a part of the OxMetrics range). In addition, there are the seasonal decomposition methods, X-13-ARIMA and Tramo-Seats, which are freely available.

A key concept in forecasting, whether for research or in practice, is to keep a hold-out sample distinct from the in-sample fitting period. Secondly, the software should permit easy incorporation of rolling origin forecasting, a critical component of forecast evaluation (see, for example, Ord, Fildes and Kourentzes, 2017 [4], for a full explanation of how these are best carried out). The revised questionnaire has included a number of queries regarding the calculation of forecast errors. In our experience, these are often poorly executed and yet are critical for the evaluation of forecasting success. For example, the user of ForecastPro can specify a hold-out sample where various error measures are calculated for both fixed and rolling origins. While AutoBox, ForecastPro and Smart are well-validated examples of software, having participated in various published competitions and therefore evaluated on standardized tests of forecasting accuracy, others may not conform to the principles of business forecasting for which we have argued (Ord, Fildes, Kourentzes, 2017 [5]). At the very least, all packages should produce forecasts of comparable accuracy to benchmarks available through R – if such comparisons have not been made, buyer beware must be the maxim.

Within the area of specialist software, the particular area of data mining (now often called predictive analytics) deserves extra attention. This software comprises a range of methods suitable for predicting individual consumer behavior. Typically, methods include logistic regression, predictive classification including CHAID and cross-sectional neural nets. The general purpose statistical packages such as SAS, Statgraphics and SPSS include these methods, but sometimes at an additional purchase cost. Weka provides free, well-regarded software; other alternatives are XLMiner, NeuroIntelligence and Neurosolutions (which may be used as Excel add-ins). Their time series capabilities are usually limited.

Econometric packages. While initially developed for econometric modeling, packages such as EViews, Oxmetrics, Stata, RATS and TSP now all include a wide range of regression-based modeling routines that can be easily applied to time series forecasting. They may also include various univariate alternatives. Other econometric packages are more specialized and focus on a limited set of routines, e.g., LINDEP, which includes a range of models suitable for classification.

Forecasting software support systems designed to support demand planning and other organizational planning activities. At their core, these systems often rely on simple smoothing methods, but their strengths are the ability to simultaneously deal with many time series and be integrated (or at least compatible) with other corporate systems such as Enterprise Resource Planning (ERP). Such all-embracing packages include SAP, Oracle, JDA and Vanguard, as well as packages focused on particular sectors of the economy such as retail and health. These packages typically cost many thousands of dollars, and their forecasting components (despite being forecasting packages in part) used to be overly limited. Now some companies such as SAP have included many advanced modeling features in their extended range of program suites including automatic regression modeling. The base suite may, however, still rely on exponential smoothing.

If they are so expensive and often limited, why do companies use them? The answer is straightforward – the packages include data handling facilities that are extremely flexible and meet the needs of most companies working in the supply chain. In particular, they will link to the ERP and financial systems. They also permit multiple-user access. The scale of the forecasting activity they embrace could be tremendous. For example, software to support Walmart’s U.S. stores may have to process more than a billion time series daily (Seaman, 2018 [7]), therefore automatic forecasting and efficient computation is essential. There are many pitfalls in purchasing such packages, not the least of which is to assume that all such packages are well-designed from a forecasting perspective. Some suppliers over-claim and, therefore, benchmarking exercises against standard validated software are critical!

A second aspect is that the design of forecasting support systems should match the organization’s processes. For example, statistical forecasts are often adjusted based on the expertise of the demand planners. This requires further analysis of the consequent errors and the “forecast value-added” arising from the adjustments.

Finally, there are mathematical and statistical programming languages such as R, Python and MATLAB, that have freely available packages and libraries that support time series analysis, econometrics and forecasting. In addition, SAS and other general-purpose statistical packages also provide programming languages that, like R, can be used to incorporate various statistical and forecasting methods into more complex analyses. The reader is referred to https://cran.r-project.org/view=TimeSeries for available packages in R related to time series analysis, modeling and forecasting.

Given that R has gained popularity over the past few years, an important criterion for evaluating any software that we ask about in the survey is its flexibility in incorporating R routines seamlessly into its design, thereby increasing its functionality. An advantage of these over commercial packages is that the authors of various innovative models and algorithms often provide open source code for the research and practice communities, ensuring state-of-the-art forecasting solutions. Furthermore, many universities train their graduates using these open source implementations, making them readily usable by the industry. This is a major paradigm shift in forecasting software.

Oliver Schaer, Robert Fildes and Ivan Svetunkov (l-r) of the Centre for Marketing Analytics and Forecasting, Lancaster University, United Kingdom.

Oliver Schaer, Robert Fildes and Ivan Svetunkov (l-r) of the Centre for Marketing Analytics and Forecasting, Lancaster University, United Kingdom.
Source: Lancaster University

Two other issues have been explored in this 2018 survey, namely forecast quality measurement and enhanced capabilities (in intermittent demand, new product methods and machine learning, as well as methods that combine the forecasts from a range of different methods). Respondents to the questionnaire were invited to highlight new developments. No consistent themes emerged; big data and machine learning were highlighted as was the need to automate the forecasting process to analyze many data series. Graphical analysis might prove particularly valuable. Readers will be able to see which packages offer these facilities.

A question easily overlooked in surveys like this is how easy the packages are to learn and use but answering would require a full hands-on review. We will note, however, that not all the programs we have used can be regarded as easy for novices. In fact, questionnaire respondents mentioned the need for packages to be usable by nonspecialists and their linkages to dashboards such as Tableau. Overall, the dimensions we have reported on here should give the reader some insight into what is available, although some of the features may be buried deep. They should also supply the practicing forecaster looking to review their software implementation with a checklist of features and a short-list of suppliers to look for in a benchmarking exercise.

Robert Fildes (r.fildes@lancaster.ac.uk) is the founding director of the Centre for Marketing Analytics and Forecasting, Lancaster University, United Kingdom. The Centre’s objective is to develop new research on the business forecasting and marketing analytics problems that organizations face and help bridge the gap between research and practice. Oliver Schaer is a doctoral candidate and research associate at the Centre. Ivan Svetunkov is an assistant professor of marketing analytics at the Centre and a developer of several R packages focusing on forecasting and analytics.

References

  1. Fry, C. and Mehrota, V., Forecasting 2016 (software Survey), OR/MS Today, June 2016, Vol.43, No. 3. Available online at http://viewer.zmags.com/publication/085442e2#/085442e2/46.
  2. Hyndman, R.J., 2017, “Forecasting functions for time series and linear models,” R package version 8.1, http://github.com/robjhyndman/forecast.
  3. Hyndman, R.J. and Khandakar. Y., 2008, ”Automatic time series forecasting: the forecast package for R,” Journal of Statistical Software, Vol. 26, No. 3, pp. 1-22.
  4. McCullough, B. D. and Heiser, D. A., 2008, “On the accuracy of statistical procedures in Microsoft Excel 2007,” Computational Statistics and Data Analysis, Vol. 52, pp. 4570-4578.
  5. Ord, K., Fildes, R., and Kourentzes, N., 2017, “Principles of Business Forecasting,” Wessex.
  6. Weller, M., and Crone, S. F., 2012, “Supply Chain Forecasting: Best Practices and Benchmarking Study,” Lancaster Centre for Forecasting White Paper. Available online: http://goo.gl/MPbAjz.
  7. Seaman, B., 2018, “Considerations of a retail forecasting practitioner,” International Journal of Forecasting. Forthcoming.

Survey directory and results

For an updated directory of forecasting software vendors who participated in this year’s survey and their complete responses, click here.

About the survey

Editor’s note: INFORMS attempted to include as many forecasting vendors and products as possible in this year’s software survey. INFORMS contacted recent survey participants, as well as any new vendors that came to our attention through the authors or our own contacts. INFORMS asked each respondent to complete an online questionnaire covering a comprehensive list of questions spanning features and capabilities, recent enhancements, licensing and fees, technical support and other areas.

The purpose of the survey is to inform the reader of what is available. The information in the survey comes directly from the vendors, and no attempt was made to verify or validate the information provided. An earlier version of this article appeared in the June 2018 issue of OR/MS Today, the membership magazine of INFORMS.

Vendors not included as of the deadline are invited to submit a completed online questionnaire, and their product and company information will be added to the online version of the forecasting survey. The survey questionnaire is available at:
https://www.informs.org/ORMS-Today/2018-OR-MS-Today-Forecasting-Software-Survey

Analytics data science news articles

Related Posts

  • 51
    Many professional and casual users do explanatory and time series forecasting in medicine, business and academia. The forecast “hits” and “misses,” particularly the latter, sometimes make headlines. The large increase in college applications for the class of 2012, especially to prestigious universities, caught some provosts by surprise. Box office returns…
    Tags: software, forecasting
  • 51
    January/February Cybersecurity: new threats, new solutions The IOT and related, hidden security risks Can analytics save U.S. healthcare system? March/April Supply chain advances and solutions Software survey: vehicle routing Capitalizing on AI & machine learning May/June Social media, marketing & analytics Real-time customer personalization Next generation revenue management July/August Software…
    Tags: software, survey, forecasting
  • 47
     NEW: Forecasting 6/9/16   Vehicle Routing - 2/17/18 Simulation - 10/8/17 Linear Programming - 6/13/17 Statistical - 2/9/17 Decision Analysis - 10/10/16 Spreadsheet Add-ins - 8/13/10 Supply Chain Management - 6/14/03 Nonlinear Programming - 6/2/98 INFORMS publishes a number of software surveys each year in conjunction with one of its…
    Tags: software, forecasting
  • 39
    Features Why optimization models fail By Patricia Randall Supply chain and manufacturing: How to avoid chaos in the field by combining simulation and real-time optimization. AI: Path to an intelligent enterprise By Joseph Byrum Imagine a future guided by artificial intelligence: Augmenting human decision-making at the enterprise level to generate…
    Tags: software, survey, forecasting
  • 37
    We were repeatedly reminded several times last year that variability can confound statistical predictions and unlikely events do occur. Upsets in sports and politics are always news, since having the underdog beat the “sure thing” is surprising and noteworthy. What is exciting in sports is unexpected in politics, since we…
    Tags: software, survey, forecasting


Headlines

Using machine learning and optimization to improve refugee integration

Andrew C. Trapp, a professor at the Foisie Business School at Worcester Polytechnic Institute (WPI), received a $320,000 National Science Foundation (NSF) grant to develop a computational tool to help humanitarian aid organizations significantly improve refugees’ chances of successfully resettling and integrating into a new country. Built upon ongoing work with an international team of computer scientists and economists, the tool integrates machine learning and optimization algorithms, along with complex computation of data, to match refugees to communities where they will find appropriate resources, including employment opportunities. Read more →

Gartner releases Healthcare Supply Chain Top 25 rankings

Gartner, Inc. has released its 10th annual Healthcare Supply Chain Top 25 ranking. The rankings recognize organizations across the healthcare value chain that demonstrate leadership in improving human life at sustainable costs. “Healthcare supply chains today face a multitude of challenges: increasing cost pressures and patient expectations, as well as the need to keep up with rapid technology advancement, to name just a few,” says Stephen Meyer, senior director at Gartner. Read more →

Meet CIMON, the first AI-powered astronaut assistant

CIMON, the world’s first artificial intelligence-enabled astronaut assistant, made its debut aboard the International Space Station. The ISS’s newest crew member, developed and built in Germany, was called into action on Nov. 15 with the command, “Wake up, CIMON!,” by German ESA astronaut Alexander Gerst, who has been living and working on the ISS since June 8. Read more →

UPCOMING ANALYTICS EVENTS

INFORMS-SPONSORED EVENTS

INFORMS Computing Society Conference
Jan. 6-8, 2019; Knoxville, Tenn.

INFORMS Conference on Business Analytics & Operations Research
April 14-16, 2019; Austin, Texas

INFORMS International Conference
June 9-12, 2019; Cancun, Mexico

INFORMS Marketing Science Conference
June 20-22; Rome, Italy

INFORMS Applied Probability Conference
July 2-4, 2019; Brisbane, Australia

INFORMS Healthcare Conference
July 27-29, 2019; Boston, Mass.

2019 INFORMS Annual Meeting
Oct. 20-23, 2019; Seattle, Wash.

Winter Simulation Conference
Dec. 8-11, 2019: National Harbor, Md.

OTHER EVENTS

Advancing the Analytics-Driven Organization
Jan. 28–31, 2019, 1 p.m.– 5 p.m. (live online)

CAP® EXAM SCHEDULE

CAP® Exam computer-based testing sites are available in 700 locations worldwide. Take the exam close to home and on your schedule:


 
For more information, go to 
https://www.certifiedanalytics.org.