Multilayer perceptron for short-term load forecasting: from global to local approach

Short-term load forecasting (STLF) aims to predict the future load demand ranging from an hour to a week ahead. This is essential for power system control and scheduling. From an energy generation point of view, STLF is necessary for electric utility operations such as unit commitment, generation dispatch, hydro scheduling, hydrothermal coordination, spinning reserve allocation, interchange and low flow evaluation, fuel allocation and network diagnosis. Since electricity load demand is the basic driver of electricity prices, load forecasting plays an important role in competitive energy markets. Forecast accuracy is a key factor in determining the financial performance of energy companies and other market participants and financial institutions operating in energy markets. Improving the STLF accuracy can significantly reduce the power system operating cost.

Neural networks (NNs) have been widely used in STLF since the early 90 s. This is due to the capacity of NNs to capture the nonlinear relationship between explanatory variables and load. Modeling this relationship is not an easy task because it is unstable in time and strongly dependent on the period of the year, day of the week, hour of the day as well as other factors. NNs have many attractive features, which are extremely useful in hard forecasting problems. These include universal approximation property, capability of learning from examples, several learning paradigms, many architectures, massive parallelism, robustness in the presence of noise and fault tolerance.

Many types of NNs have been used for STLF, such as multilayer perceptron (MLP), radial basis function NN, self-organizing map (SOM), recurrent NNs, generalized regression NNs, probabilistic NNs, adaptive resonance theory NNs, functional link NNs and counterpropagation NNs. A survey of NN applications in STLF can be found in both [1, 2] and comparison of their performances in [3]. New solutions in the field of NNs are quickly applied in STLF. For example:

STLF problems are usually decomposed to make them simpler to solve. Subproblems can be modeled using simpler NN architectures, which are easier to learn and generalize better than one big NN designed for the complex original problem. Different methods of decomposition have been proposed. In the early works on neural models for STLF, the separate models for different day types were proposed. A typical example is [12], where seven NNs were built corresponding to the 7 day types from Monday to Sunday. In [13], the load patterns were classified into weekday patterns and weekend-day patterns. The weights of NN were estimated using previous load data for each pattern. Then, the daily period was divided into three parts: 1–9, 10–19 and 20–24, and each part was modeled using individual NN. A similar approach in [14] was applied. For the forecasts of weekdays from Tuesday to Friday, there was one NN for each of these days. For Saturdays, Sundays, Mondays and days after a holiday, the daily period was divided into several parts, which were modeled by separate NNs. Decomposition into day types was also used in [15]. Separate NNs were formed for each day type, and different input variables were used. More recent examples of STLF problem decomposition into day types can be found in [16, 17, 18].

Another common practice is to decompose the load time series into 24 series, each one corresponding to an hour of the day. Then, 24 NNs with a single output learn on these series to provide load forecasts for the next day [19, 20]. In [21] for each day and each hour of the 24 h, a new model was trained on 20 previous days. Thus, this local approach produced models which are competent only for just 1 hour of the current forecasted day. For 24-h-ahead STLF in [10], 24 individual models were built based on wavelet echo state networks, each one for a specific hour of the day.

Forecasting problem decomposition into 12 months was used in [22]. The forecasting model consists of twelve NNs. Each of them performs the final 24-h-ahead load forecast for 1 month of the year. This decomposition is justified due to the differences in the weather during the year. A separate model is also designed for each month in [23]. The proposed models are composed of two hybrid NNs derived from fuzzy NNs.

Several levels of the STLF problem decomposition were used in the forecasting system known as ANNSTLF [24], which by the year 1997 was being used by over 30 utilities in the USA and Canada. This system has three MLP modules: an hourly module with 24 MLPs, a daily module with 7 MLPs and a weekly module. The final load forecast for each hour is found by a proper adaptive combination (based on the recursive least squares) of the forecasts given for this hour by the three MLP modules.

Another approach to STLF problem decomposition relies on clustering the NN inputs and designing a separate NN for each cluster. This approach combines unsupervised and supervised learning concepts. In [25], unsupervised learning is used to identify days with similar daily load and temperature patterns. The training patterns for clustering are selected among patterns from the same period of the year as the forecast day and representing the same day type. For each cluster, a NN with 24 output nodes, which produce 24 hourly loads for the forecast day, is trained. Thus, the relationship between input and output variables is modeled locally within each cluster. In [26], two-stage clustering was used. The SOM identifies similarities among the different load patterns and forms clusters. Then, the k-means algorithm identifies similarities among different clusters and groups similar clusters together. A dedicated MLP for each cluster is trained to properly forecast the load curve. Clustering is also applied on the training set in [10]. After the clusters are formulated, NNs are trained separately with the data of the corresponding clusters.

A completely different approach to the STLF decomposition problem is based on the multilevel wavelet transform (WT). A load time series consisting of both global smooth trends and sharp local variations is decomposed into generalization and details, i.e., low- and high-frequency components. These components are modeled separately by NNs. In [10], each NN gets as inputs a frequency component determined for the day d, next-day temperature and day-type index. The output of NN is the predicted next-day value of the corresponding component. NN in the next layer is used to reconstruct the next-day load forecast based on outputs (frequency components) of the NNs from the previous layer. In [27], a forecasting model composed of wavelet transform, NN and an evolutionary algorithm was proposed. Each component is predicted by a combination of NN and the evolutionary algorithm, and then by inverse WT, the hourly load forecast is obtained. The key idea of the STLF method proposed in [28] is to select the similar day’s load as the input load, apply WT to decompose it into low-frequency and high-frequency components and then use separate NNs to predict the two components of tomorrow’s load.

Among other approaches to the STLF decomposition problem, the following should be mentioned:

In this paper, we apply MLP for STLF. This type of network is the most widely used among NNs for modeling nonlinear relationships due its valuable features such as universal approximation property and flexible fitting to the target function. This flexibility is achieved easily by adding new hidden neurons. Training algorithms for MLP have been developed for many years and are still being improved. They can be implemented in many programming languages and environments such as: MATLAB, Python, R, C++, Java and Scala. There are also known effective methods of dealing with overfitting in MLP such as regularization. In the simulation study in this work, we use a powerful and robust Levenberg–Marquardt learning algorithm which interpolates between the Gauss–Newton algorithm and the method of gradient descent. To avoid overfitting, this algorithm is enriched by Bayesian regularization, which provides effective and robust MPL training.

MLP performs very well in STLF against other NNs. In [3], a comparison of several neural STLF models on four datasets is presented. The compared neural models are: MLP, radial basis function NN, generalized regression NN (GRNN), two models based on fuzzy counterpropagation NNs and three models based on self-organizing maps. MLP was trained using the local learning procedure. The forecasting results demonstrate that MLP and GRNN achieve similar level of accuracy for all datasets (with a slight advantage for GRNN). The MLP model performs very well compared to other state-of-the-art models as well as the classical statistical models: ARIMA and exponential smoothing (see comparison in Table 9 in Sect. 4).

In this work, we analyze and compare the global and local approaches to building the STLF models based on MLP. In the global approach, we built a model competent for each hour of the day and each day of the week. Local approaches include four methods of the forecasting problem decomposition:

The daily curves of the load in the proposed models are preprocessed to filter out the trend, weekly and annual cycles, and are introduced to the models as input and output variables. Depending on the approach, other input data are used: period of the year, day of the week and hour of the day. The performance of the model is dependent on how data are represented. For day of the week, six methods of coding are considered, while for hour of the day, five methods are considered. In the experimental part of this work, global and local approaches are examined as well as data representation methods.

The paper is organized as follows. In Sect. 2, data representation ways are described. The forecasting models in global and local versions are presented in Sect. 3. In Sect. 4, the models using different data representation ways are tested on real load data and compared. Finally, Sect. 5 concludes this paper.

The proposed neural models generate forecast of the load at timepoint t of the day i + τ, where i is the number of the current day and τ is the forecast horizon in days. The forecasted day is located in some period of the year and represents some day of the week. It is assumed that the load pattern of the current day i is available and can be used as an input of the model. It is represented by a vector x = [x₁x₂ … x_n], where n is a number of timepoints in a daily period (24 for hourly resolution). The load pattern of the forecasted day i + τ is represented by a vector y = [y₁y₂ … y_n]. A neural model (MLP) generates one of the y-vector component as an output: y_t. The timepoint number t, hour of the day in our case, is represented by a vector h. The forecasted day is located in the period of the year represented by a vector p, and its type (Monday, …, Sunday) is represented by a vector d. Vectors x, p, d and h can be used as inputs of NNs, and component y_t is an output. The methods of representation of input and output data are shown below.

Five variants of the forecasting models based on MLP are examined, v.1–v.5. They correspond to the method of the STLF problem decomposition. In each case, the forecasting task is to forecast the power system load at hour χ = 1, 2,…, 24 of the day i + τ.

For all cases, MLP with one hidden layer is used. The Levenberg–Marquardt algorithm with Bayesian regularization is applied for learning MLPs. In this algorithm, combination of squared errors and net weights is minimized. This expansion of the cost function to search not only for the minimal error, but also for the minimal error using the minimal weights prevents overfitting. Compared to other methods of improving generalization in NNs, Bayesian regularization gives very good results [33].

In this section, the neural models v.1–v.5 as well as data representation methods are evaluated on real data: four data sets containing hourly loads of Polish (PL), British (GB), French (FR) and German (DE) power systems in the period 2012–2015. (The source of data is www.​entsoe.​eu.) A one-day-ahead STLF problem is considered (τ = 1). Then, the forecast accuracy of the best variant of the proposed neural model is compared with levels of accuracy achieved by state-of-the-art models applied to STLF. The comparative models include neural networks, linear regression, nonparametric regression, clustering-based models, artificial immune systems, ARIMA, exponential smoothing and the naïve model.

The optimization and training procedures for neural models in variants v.1–v.4 are as follows. First, the number of hidden neurons and the best methods of input data representation are selected. To do this, the training is repeated for each variant of data representation as well as for #neurons = 1, 2,…, g. For example, in the case of model v.1, we have four input variables: x_i, p_i+τ, d_i+τ, h_t. The day of the week is encoded using one of six ways, and the hour of the day is encoded using one of five ways. Thus, there are 5 × 6 = 30 combinations of input data representation. For each combination, we train the model composed of 1, 2,…, g neurons. The training set is created using data from the period 2012–2013. After training, the model is tested on data from 2014. The best model is selected having the lowest error on the test period. Then, the best model is trained on data from the period 2012–2014 and then tested on data from 2015. The mean error for the forecasting tasks from the test period (2015) is a measure of the model quality.

In the case of the neural model in variant v.5, the optimization and training procedures are different. This model is trained for each forecasting task from the test period (2015) independently. First, to select the number of hidden neurons, ten models are built for ten forecasting tasks from the history, which are similar to the current forecasting task. The history from which these ten tasks are selected is limited to the period covering year 2014 and the period of 2015 preceding the current forecasting task for the day q. By similar forecasting tasks to the current one, we mean tasks for the same day-type δ as the current task and having x-vectors similar to the x-vector of the current task (x_q). The similarity measure between x-vectors is the Euclidean distance. For example, when the forecasting task concerns July 1, 2015, a Wednesday, we select ten similar forecasting tasks from the period from January 1, 2014, to June 29, 2015. For PL data ,these selected tasks are (ranked in the order of similarity): June 11, 2014, June 25, 2014, June 17, 2015, June 24, 2015, July 2, 2014, July 16, 2014, July 9, 2014, June 18, 2014, July 30, 2014 and June 10, 2015. As we can see, these days are from the same period of the year as the current forecasted day. The model learns for each of the ten similar tasks independently on the training set generated from the historical data according to (13), where now q is the similar day number. The training is repeated for #neurons = 1, 2,…, g, and the optimal number of neurons is selected for which the mean error determined on similar tasks is minimal. Then, the optimal model learns on the training set (13) generated from the period starting on January 1, 2012, and the forecast for the current task is generated. Thus, for each forecasting task from 2015, a separate NN is created and optimized on the historical forecasting tasks which are most similar to the current one.

The error measure applied in this study is the mean absolute percentage error (MAPE), which is traditionally used as an error measure in STLF. Atypical days such as public holidays are excluded from the training and test sets (between 10 and 20 days in a year).

The errors for the model v.1 with different methods of encoding of the day type (dc) and hour of the day (hc) are shown in Fig. 1. Note that in most cases binary coding for the hour of the day provides much better results than other ways of coding. In the case of PL and FR data, we get the best results when hour of the day is coded on 24 bits and the day of the week is coded using sine and cosine functions. For GB data, best results are obtained when the hour of the day is coded using natural binary code and the day type is coded on seven bits. The best results for DE data are achieved when the hour of the day is represented in natural binary code, and the day type is encoded in two bits representing one of four groups (dc = 6). The best methods of data representation are summarized in Table 3, which also shows the optimal numbers of hidden neurons and errors for 2014 and 2015. The error for 2015, as an error on data unseen during the learning and optimization procedures, is the right measure of the model’s performance.

Table 4 shows results for the model in variant v.2. Note that in optimization procedures in all cases, binary coding was selected for the hour of the day. The forecasts for Mondays are usually less accurate than forecasts for other days of the week. Unexpectedly, mean errors generated by model v.2 are higher than errors generated by model v.1 for all datasets. This leads to the conclusion that the decomposition of the forecasting problem on separate models for each day of the week is not a good idea.

Results for the models in variants v.3 and v.4 are presented in Tables 5 and 6, respectively. Note that the optimal number of neurons for v.4 is about 2 or 3 on average and for v.3 about 5–7. This is because models v.4 are “more local” than v.3, and the relationships between input and output variables are easier and can be modeled by simpler NN. The global approach v.1 needs 8–14 neurons, and variant v.2, which is decomposed on 7 NNs needs 6–10 neurons on average. Thus, we can expect that the “most local” approach v.5 will need even fewer neurons than v.4. In Table 7, errors are shown for models v.5 in three variants:

The best results for each dataset in Table 7 are marked with an asterisk (best results were confirmed by Wilcoxon rank sum test with 5% level of significance). As we can see from this table, there is no difference in errors between variants v.5a and v.5c. But note that variant v.5c does not require the time-consuming optimization phase.

A comparison of the proposed approaches v.1–v.5 is given in Table 8. The lowest errors are for PL data, then for DE and FR data. In contrast, errors for GB data are up to two times higher than for the other data sets. As in Table 7, asterisks indicate best results, which have been statistically confirmed. The most accurate approaches are v.5 and v.4. These approaches do not use the day of the week and hour of the day as inputs which means that searching for the best way of representation is unnecessary for these models. In variant v.5c, searching for the optimal number of neurons is unnecessary as well, meaning the model in this variant can be built very fast.

In Fig. 2, errors for each hour of the day and day of the week are shown. In the case of PL, FR and DE data, the most outlying errors are for Mondays. Errors for the earliest hours of the day are the lowest, while for hours 12–18, corresponding to the peak load, they are highest.

Finally, we compare the forecast accuracy of the best variant of the proposed neural model, v.5, with the accuracy of the state-of-the-art models applied to STLF. The models were tested in one-day-ahead STLF problems on the following datasets:

The comparative models are:

The first seven models are based on different types of NNs and were described in detail in [3]. The next two models, PCR and PLSR [35], are linear regression models in which the components of the input patterns are constructed by the linear combination of the original components. In these models, the relationship between input and output patterns is modeled locally in the neighborhood of a query pattern. N-WE and FNM are nonparametric regression models [36], where the regression curve is a linear combination of y-vectors weighted by the function which nonlinearly maps the distance between x-vectors. FP1 + k-means and FP2 + k-means [36] aggregate the x- and y-patterns into clusters, assign the query pattern to the cluster and reconstruct the forecasted y-pattern from the cluster characteristics. Artificial immune systems, AIS1, AIS2 and AISLFS [36, 37], are biologically inspired computation methods, where the forecasting problem is solved in the immune memory creation process. In these models, antibodies are the recognition and prediction units, which memorize features of the time series and reconstruct the forecasted pattern. For ARIMA and ES, the time series were decomposed for each hour of the day (or half hour, depending on the time series resolution) and a separate series was created. In this way, a daily seasonality was eliminated. ARIMA or ES was used for the independent modeling of these series. In the above list of forecasting models, except ARIMA, ES and Naïve, the time series are represented by patterns defined in the same way as in this work [see Eqs. (1) and (2)].

Table 9 gives the forecast errors for the test period (last year of the time series) for the models listed above. The lowest errors are in bold. As we can see from this table, MLP v.5 is among the most accurate models. It should also be noted that, in most cases, pattern-based methods outperform the classical methods such as ARIMA and ES.

The main contribution of this work is to examine global and local versions of the neural models for STLF. The models are analyzed in the context of data representation methods. Daily load curve is introduced to the models as an x-vector: the normalized vector of the hourly loads. This preprocessing simplifies the forecasting problem by filtering out both the trend and seasonal variations of periods longer than a day. X-vectors express the shape of the daily curve. Similar preprocessing is used for an output variable. The day of the week is encoded in six ways and the hour of the day in five ways. In optimization procedures, the best ways of coding as well as the number of hidden neurons are selected for each forecasting model.

In a global approach, the model is competent for each day of the week and each hour of the day. The relationship between input and output variables is complex in this case, which means a more complex network with more hidden neurons is required. Both the learning and optimization of such model are difficult, time-consuming tasks. While the accuracy of this model is limited, the decomposition of the forecasting problem into subproblems, and modeling these subproblems separately, should lead to an improvement in accuracy. The first decomposition method relies upon splitting the problem into seven subproblems, one for each day of the week individually. However, an experimental study did not confirm better results for this approach. The second decomposition method splits the problem into 24 subproblems, one for each hour of the day. Local NNs are built for each hour and results are improved. Further improvement is achieved when the problem is decomposed into subproblems representing each day of the week and each hour of the day. In this case, the local relationships between the input and output variables within the subproblems are simpler and can be modeled using less neurons in easy optimization and learning procedures. Finally, the most local decomposition method splits the problem into separate forecasting tasks, i.e., forecasts for a given hour of a given day. In this case, an individual NN learns for each forecasting task and is competent only for this task. New tasks require new NN learning. An advantage of this method is that the model does not require an optimization phase (selection of the number of neurons and data coding method). In its optimal variant, it only has two hidden neurons, so, its learning is very fast. The most local models, v.4 and v.5, reduced the forecast errors significantly, when compared to the global model.

The final recommendation for STLF is using the local MLP models v.4 or v.5 due to the most accurate results. Note that these models have very simple architecture: in v.4 only 2–4 hidden neurons are needed, and in v.5 just two or even one hidden neuron provides sufficiently accurate forecasts. Such simple models learn much faster and are more resistant to overfitting than more complex models v.1, v.2 and v.3. Moreover, the landscape of the error function for them is less complex, so finding a global minimum is more likely.