Forecasting with Artificial Intelligence

Artificial Intelligence (AI), as its name implies, is another form of intelligence analogous to our own Human Intelligence (HI). There is growing agreement that based on advances in deep reinforcement learning, AI has surpassed HI in some specific areas such as games and image/speech recognition. At the same time, there is disagreement among leading experts whether, and how quickly, this dominance will spread to other areas, what the implications of AI dominance are, whether AGI (Artificial General Intelligence) is feasible, let alone imminent, and on the possibility of exploiting advances in AI and related technologies to augment our own intelligence. The two authors have been debating whether AI is approaching HI or is closer to that of a hedgehog, and present their common position in this chapter that starts by comparing AI’s achievements to those of HI, concluding that they are of a different, complementary nature, while also presenting its current capabilities and future challenges. The second part discusses the future of AI and the uncertainties and challenges it faces. The final part presents four possible scenarios and discusses Intelligence Augmentation (IA) and the possibility of substantial improvements in HI by exploiting progress in AI and advances in the related technologies of nanotechnology and neuroscience. There is a concluding section deliberating about the long-term future of AI by presenting the views of the two authors.

This article explores how expectations concerning the future of artificial intelligence (AI) might shape economic decision-making. The article discusses the possibility that individuals may save less in anticipation of either extreme wealth or catastrophic outcomes resulting from the development of AI. Additionally, young people may choose to invest less in education if they believe that AI will make any acquired human capital obsolete before they reach middle age. This article also investigates the possibility of conflict between the US and China arising from expectations about the military value of AI and the importance of economies of scale in AI development. The possibility that politicians may increase borrowing in anticipation of an AI-related economic boom is also explored.

Time series forecasting covers a wide range of methods extending from exponential smoothing and ARIMA models to sophisticated machine learning ones, such as neural networks and regression-tree-based techniques. More recently, deep learning methods have also shown considerable improvements in many forecasting applications. This chapter provides an overview of the key advances that have occurred per class of method in the last decades, presents their advantages and drawbacks, describes the conditions they are expected to perform better under, and discusses some approaches that can be exploited to improve their accuracy. Finally, some directions for future research are proposed to further improve their accuracy and applicability.

Forecasting the demand for new products is crucial given the level of investment required for a launch. It is also challenging and risky in an environment of vigorous economic competition, evolving customer expectations, and the emergence of new technologies and innovations. Given the high failure rate of new launches (70–80 percent for consumer-packaged goods), the accuracy of demand forecasts is a top priority for decision-makers. Underpredicting demand leads to a loss of potential sales; overpredicting it leads to costly excess inventory.

Forecasting new product demand has traditionally been done using a variety of techniques: judgmental methods, market research like surveys of buyers’ intentions, market testing, expert opinion methods like the Delphi method, diffusion models like the Bass model, and statistical modeling through a variety of time series and/or multivariate techniques. More recently, machine learning has been added to the mix. The selection depends somewhat on whether the new product is: (a) new to the world, (b) new to the firm, (c) an addition to existing product lines, or (d) an improvement or revision to existing products.

Machine learning is a good candidate when we have lots of data, including the sales history, on existing products that are similar to the new one. Although humans use this approach too, the idea is that machine learning should be able to do it faster and more accurately. Many papers and case studies are available on using machine learning to forecast existing products with historical data. However, when it comes to new products with little or no history, the literature is very limited.

In this chapter, we will review the main techniques for predicting new product demand, focusing on machine learning. We also review four recent case studies that confirm that machine learning can improve accuracy of demand forecasts for new products.

Forecasting models that are trained across sets of many time series, known as global forecasting models, have recently shown promising results in prestigious forecasting competitions and real-world applications, outperforming many state-of-the-art univariate forecasting techniques. This chapter provides insights on why global models are important for forecasting in the context of Big Data and how these models outperform traditional univariate models, in the presence of large collections of related time series. Furthermore, we explain the data preparation steps of global model fitting and provide a brief history of the evolution of global models over the past few years. We also cover the recent theoretical discussions and intuitions around global models and share a summary of open-source frameworks available to implement global models.

This chapter shows how to use large quantities of data for improving forecasting accuracy of Artificial Intelligence models. Time series forecasting usually proceeds by fitting a model to a given time series. Because time series usually have just a few observations, and AI models require considerable amount of data to train, forecasting by fitting an AI model to each time series in isolation tends to underperform when compared to forecasting with simple models. In this chapter, we analyze a methodology that overcomes the data limitations of AI models in forecasting, therefore enabling the application of the full range of AI techniques to forecasting, which will lead to improvements in predictive accuracy. The methodology, called cross-learning, proceeds by training a single AI model across multiple time series, as opposed to local learning, that trains a model on each time series. We show how cross-learning is universally applicable to arbitrary datasets as long as complex models are used, despite its seemingly strong restrictions. We analyze the statistical aspects of cross-learning compared to local learning, illustrating how cross-learning benefits from forecasting many time series and providing guidance on how models and datasets can be tuned in order to maximize accuracy. We give explicit illustrations of how notable time series processes such as logistic growths, polynomial trends or periodic patterns have an equivalent, perfect representation as complex autoregressive processes that are learned automatically by training them on whole datasets. We show an additional benefit of cross-learning in how it can transfer information across series in the dataset, when some time series can be considered to have information that could be helpful to forecast other time series in the dataset. A real case of COVID-19 predictions is analyzed to exemplify this transfer of information mechanism. The guidelines we provide are model-agnostic, they either motivate or are directly applicable to the range of AI models currently available.

Machine learning (ML) based time series forecasting models often require and assume certain degrees of stationarity in the data when producing forecasts. However, in many real-world situations, the data distributions are not stationary and they can change over time while reducing the accuracy of the forecasting models, which in the ML literature is known as concept drift. Handling concept drift in forecasting is essential for many ML methods in use nowadays, however, the prior work only proposes methods to handle concept drift in the classification domain. To fill this gap, we explore concept drift handling methods in particular for Global Forecasting Models (GFM) which recently have gained popularity in the forecasting domain. We propose two new concept drift handling methods, namely Error Contribution Weighting (ECW) and Gradient Descent Weighting (GDW), based on a continuous adaptive weighting concept. These methods use two forecasting models which are separately trained with the most recent series and all series, and finally, the weighted average of the forecasts provided by the two models is considered as the final forecasts. Using LightGBM as the underlying base learner, in our evaluation on three simulated datasets, the proposed models achieve significantly higher accuracy than a set of statistical benchmarks and LightGBM baselines across four evaluation metrics.

Forecast combinations are considered a standard practice in many time series forecasting tasks, due to their documented success in improving the accuracy and robustness of the final forecasts. Regardless of the chosen combination scheme, constructing an ensemble of models reduces the impact of individual models’ biases and the need for selecting a single best model. These potential benefits are even more critical when forecasting neural networks are involved, as their use introduces new challenges, mainly linked with their stochastic nature and the large number of hyper-parameters influencing their performance. Motivated by the widespread adoption of neural networks in forecasting applications, in this paper we explore in greater detail the combination of forecasts produced by ensembles of feed-forward networks. We focus on the impact that different initialization seeds and “high-level” parameters, such as the size of the input vector and the loss function, have on forecasting accuracy. We empirically evaluate the performance of individual models and ensembles of models, using three sets of series from the M4 competition. Our results suggest that ensembling neural networks significantly boosts forecasting performance, but at the cost of additional computational time.

Many industrial applications concern the forecasting of large numbers of time series. In such circumstances, selecting a proper prediction model for a time series can no longer depend on the forecaster's experience. The interest in time series forecasting with meta-learning has been growing in recent years, as it is a promising method for automatic forecasting model selection and combination. In this chapter, we briefly review the current development of meta-learning methods in time series forecasting, summarize a general meta-learning framework for time series forecasting, and discuss the key elements of establishing an effective meta-learning system. We then introduce a meta-learning python library named ‘tsfmeta’, which aims to make meta-learning available for researchers and time series forecasting practitioners in a unified, easy-to-use framework. The experimental evaluation of the ‘tsfmeta’ on two open-source datasets further shows the promising performance of meta-learning on time series forecasting in various disciplines. We also offer suggestions for further academic research in time series forecasting with meta-learning.

In economics and many other forecasting domains, the real world problems are too complex for a single model that assumes a specific data generation process. The forecasting performance of different methods changes depending on the nature of the time series. When forecasting large collections of time series, two lines of approaches have been developed using time series features, namely feature-based model selection and feature-based model combination. This chapter discusses the state-of-the-art feature-based methods, with reference to open-source software implementations.

Demand forecasting in the online fashion industry is particularly amendable to global, data-driven forecasting models because of the industry’s set of particular challenges. These include the volume of data, the irregularity, the high amount of turn-over in the catalogue and the fixed inventory assumption. While standard deep learning forecasting approaches cater for many of these, the fixed inventory assumption requires a special treatment via controlling the relationship between price and demand closely. In this case study, we describe the data and our modelling approach for this forecasting problem in detail and present empirical results that highlight the effectiveness of our approach.

Forecasting and optimisation are two major fields of operations research that are utilised to deal with uncertainties and to make the best decisions. These methods are widely used in academia and practice and have contributed to each other growth in several ways. These methods can be used together to solve various problems in transportation, scheduling, production planning, and energy where both forecasting and optimisation are needed. However, the nature of the relationship between these two methods and how they can be integrated for better performance have not been explored or understood enough. We advocate the integration of these two methods and explore several problems that require both forecasting and optimisation. I will investigate some of the methodologies that lie at the intersection of machine learning with forecasting and optimisation to address real-world problems. I will provide several research directions and use cases for researchers and practitioners interested to explore this interesting arena.

This chapter presents a novel method in monetary policy analysis and inflation forecasting. The author presents a hybrid model, which imposes different economy dynamics in different periods of the Business Cycle—LSTVAR-ANN. LSTVAR-ANN provides a plethora of insights such as impact of monetary policy in expansion or recession periods, components of the Business Cycle or inflation forecasts. The research was conducted on US data. In LSTVAR-ANN model regimes are defined by a smooth, continuous transition function. The output of the transition function can be interpreted as a metric of the Business Cycle momentum. In this research, the author used Index of Customer Confidence of University of Michigan as a proxy. ANN part of the model helps to “observe“ consumer confidence via Internet search data (here: Google Trends). This chapter answers three questions stated by the author: is it possible to observe consumer confidence (thus the Business Cycle) using Internet searches? Does monetary policy affect prices differently in different business cycle periods? Does differentiation of regimes enhance inflation forecasts?

The last decade has been an exciting and fruitful time for the advancement of forecasting. Traditional time series methods have been enhanced, and in some cases supplanted, by a new generation of data scientists bringing new approaches from machine learning and artificial intelligence. But this rapid innovation has fueled claims of performance improvement that require proper assessment. The forecast value added (FVA) framework provides an alternative to traditional methods for assessing forecasting performance.