Relationship Between Calibration Time and Final Performance of Conceptual Rainfall-Runoff Models

Although relations between elements of the hydrological cycle are well understood, their quantitative description is still a challenge. Many hydrologists are interested in processes that describe a relation between precipitation and runoff at the catchment scale. However, depending on many meteorological, land cover and soil-related factors that are variable at small spatial and temporal scales, the quantitative description of the processes that relates precipitation to runoff at the catchment is difficult. It is frequently assumed that models which could be used in practice cannot be too detailed (Bergström 1991) and must roughly approximate many physical processes that occur in the catchment-scale environment – such models are frequently called “conceptual” rainfall-runoff models. If the models simplify all processes at the catchment scale, they are called “lumped”. It turns out that for rainfall-runoff modelling lumped conceptual models are considered to perform not worse than more complex distributed models (Vansteenkiste et al. 2014; Lobligeois et al. 2014), even though they are vulnerable to temporal resolution (Jie et al. 2018) and variability of hydro-meteorological conditions (Poncelet et al. 2017). Although conceptual models are much simpler and require much less information than physical ones, they still have some theoretical background, contrary to purely data based-models like artificial neural networks, stochastic transfer functions or nearest neighbourhood search approaches (Pechlivanidis et al. 2011). However, irrespective of which model is used, it needs to be calibrated. Even if model parameters have a physical representation, they almost never can be precisely measured (Beven 2012). The problem of rainfall-runoff model calibration, considering various sources of uncertainty or not, has been discussed in various reviews (Beven 2012; Refsgaard 1997). In this study, attention is focused on single-objective automatic Swarm Intelligence or Evolutionary Algorithms that aim at finding the best set of parameters for a given model and catchment and become very popular in water-related studies (Tayfur 2017). Even though it is well known that due to the concept of equifinality (Beven 2012) the solution found would strongly depend on the model representation, objective function or data, and would not have “general” meaning (Vrugt et al. 2008; Merz et al. 2011; Osuch et al. 2015), such approach to model calibration remains widespread, as discussed below.

During the last decade a number of papers aimed at comparison among optimization algorithms applied to rainfall-runoff model calibration have been published (Goswami and O’Connor 2007; Arsenault et al. 2014, Piotrowski et al. 2017a). Such studies generally showed that many algorithms perform similarly well and no best method may be determined. However, the performance of optimization algorithm depends largely on the number of allowed function calls (or model runs). The number of function calls is a measure of method’s computational time independent of the code, computer or language used. As shown in two recent methodological papers (Posik et al. 2012; Piotrowski et al. 2017b) the ranking of optimization algorithms highly depends on the number of available function calls. This is inevitably linked with the problem of computational efficiency. Unfortunately, in hydrological modelling the problem of proper setting the number of function calls is rarely discussed. In many papers, the number of model runs used by calibration procedure is even not clearly stated. If the number of function calls is given in the paper, values used may vary severely. For example, a million function calls were used to calibrate air-to-stream temperature models with four to eight parameters by means of Particle Swarm Optimization (PSO) in Toffolon and Piccolroaz (2015), what seems a large exaggeration. As many as five million function calls are used in Bi et al. (2016) for water distribution system optimization problems, however, this time such large number could be justified by the fact that even 1000-dimensional problems were tackled. On the other hand, Wang et al. (2010) set the number of function calls to only 100 when using a simple Genetic Algorithm and Shuffled Complex Evolution (SCE-UA) method for calibration of grid-based distributed rainfall–runoff model with four parameters. Such examples show how large differences may be spotted in the literature, often without any justification.

In some hydrological studies, the relation between the performance of the conceptual rainfall-runoff model and the number of function calls is considered, often referring to the graphically-illustrated convergence speed (Tolson and Shoemaker 2007; Arsenault et al. 2014; Piotrowski et al. 2017a). In such figures the relation between the quality measure of the best solution found so far is plotted against the already performed number of function calls. However, from such illustrations published in various studies aiming at rainfall-runoff modeling different conclusions could be drawn. From Arsenault et al. (2014) one may learn that some algorithms initially converge quicker than the others, but such methods often perform worse than the best ones at the end of the search. A similar result was obtained by Jeon et al. (2014), but for another goal, namely calibration of long-term hydrologic impact assessment model. On the contrary, although in Tolson and Shoemaker (2007) and Piotrowski et al. (2017a) also differences in convergence speed between various algorithms applied for calibration of conceptual rainfall-runoff models are noted, finally vast majority of methods reach almost equal performance. This may suggest that most “good” methods may lead to the solutions of equal quality if the number of function calls is large enough, what again puts attention to both the concepts of equifinality (Beven 2012) and algorithm efficiency.

In Arsenault et al. (2014) study one may also find another approach to this topic. It shows after how many function calls the particular algorithm on average reaches the 95% of the best value of the Nash-Sutcliffe coefficient (Nash and Sutcliffe 1970) found during the whole search (e.g. 25,000 function calls). It turns out that such close-to-the-best results are often found during the first half of the search, and in about 50% of cases – during the first 5000 function calls. Hence, is the longer search just a waste of time?

Studies discussed above addressed the problem of how fast the particular method converges when the predefined number of allowed function calls is preset and anyway used. Would such results be similar to those obtained when different maximum numbers of function calls are assumed? For example, when setting the maximum number of function calls to 20,000, Jeon et al. (2014) found that the variant of Genetic Algorithm they apply perform better than SCE-UA method (Duan et al. 1992) until 5150 function evaluations is used, but SCE-UA is better afterwards. Does it means that when one sets the maximum number of function calls to 5000, Genetic Algorithm would still perform better? Here two points have to be addressed.

First, best solution found so far by the algorithm cannot deteriorate during a run. However, if a number of runs is performed with some number of function calls, and then the same number of runs is repeated independently with a larger number of function calls, during repeated runs better solution may be found sooner, or only worse solutions may be found. Hence, if someone performs 100 runs for, say, 10,000 function calls, never the average performance after 5000 calls would be better than those after 10,000 calls (and almost always solutions found after 10,000 calls will be better). However, if one performs 100 runs with 5000 function calls each, and then independently performs 100 runs with 10,000 function calls each, it may happen that the average performance found when 5000 function calls were allowed turns out better than in the case when the number of function calls was set to 10,000. This may even be not rare for problems on which algorithms are easily trapped in a local optimum, hence additional function calls are simply wasted.

Second, if the optimization algorithm is developed in a “smart” way, its behaviour at a particular stage of the run would differ, depending on the remaining computational budget. Such algorithms search differently when the number of allowed function calls is low then when it is high. For example, in variants of PSO with inertia weight (Shi and Eberhart 1998), the inertia weight is decreasing linearly with time such that the lower the maximum number of function calls is, the quicker the decrease of inertia weight. In Successful-history based Differential Evolution with linear population size reduction (L-SHADE, Tanabe and Fukunaga 2014), a winner of IEEE Conference on Evolutionary Computation (IEEE CEC) in 2014, population size is linearly reduced with the number of function calls from 18∙D (where D is the problem dimensionality) at the beginning to just four individuals at the end of the search. Also, Dynamically Dimensioned Search (Tolson and Shoemaker 2007), developed for hydrological applications, has been designed in order to quicken the search when the computational budget is low and performs more explorations when it is high. It results in different behavior, and hence relatively small differences in final results achieved, when the preset number of function calls varied between 200 and 2000 (see Tolson and Shoemaker 2007).

Hence, to verify how many function calls are needed to solve efficiently the problem of conceptual rainfall-runoff calibration when advanced Evolutionary Algorithms are used, instead of referring to plots showing convergence speed, independent tests must be performed with different numbers of function calls. Moreover, it needs to be verified whether any relation found between the number of function calls used and the model performance holds not only for the data used for calibration but also for validation ones. This is the main goal of the present study, in which two lumped conceptual rainfall-runoff models, HBV (Bergström 1976; Lindström et al. 1997) and GR4J (Perrin et al. 2003) with snow module (Piotrowski et al. 2017a), are calibrated by means of three optimization algorithms with four different maximum numbers of function calls, set to 1000, 3000, 10,000 and 30,000 (each about three times larger than the previous one). The aim of this paper is to find a relationship between the modeling performance and the number of function calls when the advanced optimization procedures are used.

In addition to the main goal, differences between the perception of the same results depending on the hydrological criterion used are emphasized. Among a few criteria applied in the paper, we put attention to two specific ones, namely Persistence Index (PI) (Kitanidis and Bras 1980) and Nash-Sutcliff coefficient (NSC) (Nash and Sutcliffe 1970). According to both PI and NSC criteria, the best fit is obtained when NSC or PI = 1. However, each criterion measures something else. Using the model is a better choice than assuming that the forecasted runoff will be the same as the recent observation in the case of PI, or the same as long-term mean in the case of NSC, as long as the value of the criterion is above 0. As both criteria are frequently used, this paper researches how much they may differ in practice.

In this study two lumped conceptual rainfall-runoff models, HBV and GR4J, are used. For the description of both models, see main source papers: Bergström (1976) and Lindström et al. (1997) in case of the HBV and Perrin et al. (2003) in case of the GR4J. A specific version of the HBV (with 13 parameters) and the GR4J with snow module (with seven parameters, four from the basic GR4J and three from snow module) that are used in this study are discussed in Piotrowski et al. (2017a). Both models have found numerous practical applications (Lindström et al. 1997; Beven 2012; Tian et al. 2013).

Both models are applied at the daily time scale. The river runoff (y_t + 1) at time t + 1 is simulated based on the precipitation (R_t), air temperature (T_t) and evapotranspiration (E_t) (related to air temperature according to Hamon’s (1961) method) data from previous day t. Hydrological data are always collected from the single gauge station in this study, but, depending on the catchment considered, the meteorological data may come from a single meteorological station located within the catchment, or a few different stations located within, or close to the catchment. Time series from each catchment are divided into the calibration and the validation sets. Calibration set, composed of roughly 70% of available data, is used during model optimization; the validation set is used only to verify the quality of calibrated models on the independent data. However, the first year (365 days) of the calibration data is considered as a warm-up period and is not used to compute the value of the objective function.

In this study, runoff values simulated by HBV or GR4J models are either considered directly (such results are called “raw” further in the paper), or after applying error correction procedure (Refsgaard 1997; Madsen and Skotner 2005; Liu et al. 2016). In the second case, after termination of the calibration procedure, the raw results from the HBV and the GR4J models are updated by means of linear regression with exogenous inputs; in this version, the past forecasts from the raw HBV or the raw GR4J predictions are added as exogenous inputs to the linear regression error model:where m (model) denotes the HBV or the GR4J and \( {\varepsilon}_t^s={y}_t-{y}_t^m \) is the prediction error. The final simulated runoff is calculated as \( {y}_{t+1}^s={y}_{t+1}^m+{\varepsilon}_{t+1}^s \). Here δ is set to 3 days for both the HBV and the GR4J.

Five catchments are considered in this study: the lowland Suprasl (Poland) and the Irondequoit Creek (New York state, USA), the hilly Fanno Creek (Oregon, USA), and the mountainous Biala Tarnowska (Poland) and the Cedar River (Washington state, USA). Their basic characteristics and descriptions are given in Table 1 (data sources for USA catchments are: US Geological Survey (USGS) and National Centers for Environmental Information, National Oceanic and Atmospheric Administration (NOAA); for Polish catchments: Institute of Meteorology and Water Management (IMGW)). Table 1 also includes detailed, site-specific information on data sets used, gaps in data, and splitting observation series into the calibration and validation sets.

In case of the Biala Tarnowska and the Cedar River catchments, the lumped precipitation and air temperature data series are obtained by Thiessen Polygons (Thiessen and Alter 1911) method from measurements collected at three and five stations, respectively (see Table 1). In case of the Suprasl River, the air temperature data come from Bialystok station only, but lumped precipitation data set is based on measurements from five meteorological stations. For the Fanno Creek and the Irondequoit Creek, all meteorological data come from a single meteorological station.

Using just one calibration method could bias the results. Hence, calibration of each model for every catchment and with each considered number of function calls is performed by three optimization algorithms: Modified Differential Evolution with p-best crossover (MDE_pBX, Islam et al. 2012), Successful parents selecting L-SHADE with eigenvector-based crossover (SPS-L-SHADE-EIG, Guo et al. 2015) and Genetic Learning Particle Swarm Optimization (GLPSO, Gong et al. 2016). Control parameters of all three optimization algorithms are set as suggested in the source papers (in the case of SPS-L-SHADE-EIG, the so called “default” variant of control parameter settings has been used, see Guo et al. 2015). Population size has been set to 100 for MDE_pBX (Islam et al. 2012; Piotrowski 2017), 50 for GLPSO (Gong et al. 2016) and follows linear reduction scheme from 19D (where D is a problem dimensionality; D = 13 for the HBV and 7 for the GR4J) to 4 in SPS-L-SHADE-EIG (Guo et al. 2015). Note that in SPS-L-SHADE-EIG algorithm the values of population size are related to the computational budget, hence the method will behave differently when large and small numbers of maximum function calls are preset.

To verify the impact of the number of allowed function calls, all calibration experiments are repeated with four different values of function calls, set to 1000, 3000, 10,000 and 30,000. The idea is that in various tests the numbers of function calls differ roughly by a factor of three. Tests are independent, what is especially important for SPS-L-SHADE-EIG that scale population size with the remaining number of function calls.

In this study, all tests are repeated 30 times, each time with different, randomly generated initial populations. This gives a sample of 30 solutions for each considered variant (“variant” means, for example, calibration of the raw HBV model by means of GLPSO for the Cedar River with the maximum number of function calls set to 30,000). The total number of variants considered in this paper is 240: 2 models × 2 error correction procedure (used or not) × 3 optimization algorithms × 4 maximum numbers of function calls × 5 catchments.

Three criteria are considered in this study:

where \( {y}_t^s \) is a simulated runoff for time t, y_t is a measured value of runoff, and N is the number of daily data in the particular data set (calibration or validation) for particular catchment;

where \( \overline{y_t} \) is a mean of N measured runoff values;

where L is a lead time, set to 1 in this study.

To compare the results the 30-runs mean values of every criterion for each variant are computed, together with associated standard deviations. The MSE is used for calibration as the objective function, NCS and PI are computed after each model is calibrated.

Detailed results, including 30-run mean and standard deviation values of MSE, PI and NSC criteria for all tested variants, are given in Supplementary Tables 1–10 that are available online as Supplementary Material. To facilitate reading, the relation between the number of function calls and the model performance is presented in Figs. 1, 2, 3, 4 and 5, where values of MSE, PI, and NSC obtained for both the calibration and the validation data are shown. Due to space limitation, the results obtained with the use of error correction procedure are illustrated graphically only for two selected catchments (Fanno Creek, Fig. 1, and Irondequoit Creek, Fig. 2); results for all catchments are given in Supplementary Tables 1–10.

This paper aims at studying the impact of the assumed number of function calls to be used during calibration of the lumped conceptual rainfall-runoff model on the final performance. Tests with different numbers of function calls (1000, 3000, 10,000 and 30,000) are performed independently, hence longer calibration does not necessarily imply better results. Two models are tested (HBV and GR4J), each applied with or without error correction procedure (Madsen and Skotner 2005), and with three calibration procedures (GLPSO, MDE_pBX and SPS-L-SHADE-EIG) at five catchments (the mountainous Biala Tarnowska, Poland and Cedar River, WA, USA; the hilly Fanno Creek, OR, USA; the lowland Irondequoit Creek, NY, USA and Suprasl, Poland) located in temperate climatic conditions. Research is based on 14–39 years long daily data that are divided into calibration and validation parts.

For various catchments, substantial differences in modelling performances for the calibration and the validation data may be observed, what is in agreement with Merz et al. (2011) and Osuch et al. (2015). Such results have large implication for the number of function calls that should efficiently be used during conceptual rainfall-runoff model calibration. For the calibration data set, often the more function calls are used, the better results are obtained. However, differences between results obtained when 10,000 and 30,000 function calls are used are often meaningless. For validation data, this is often not the case. Depending on the catchment and methodological details, the best results for validation data may be obtained with any considered number of function calls, from 1000 to 30,000. Sometimes the longer optimization is, the better results are obtained for the calibration set, but the poorer for the validation one.

Such lack of consistent relation between the rainfall-runoff model performance and the length of model calibration may clarify the total mess in the hydrological literature, where some models are calibrated for hundreds, other for millions of function calls. As no rules may be observed, each time practitioners must look for the best setting for their specific application. Hence, when calibrating conceptual rainfall-runoff models for practical applications, tests with at least two different numbers of function calls, one very small and one moderate (e.g. 1000 and 10,000) are advised, to see if the choice affect the results; in each case calibration should be performed at least a few times to verify consistency of the results.

In the future, it is recommended to verify how the choice of the stopping conditions of Markov Chain Monte Carlo (MCMC) methods (see Vrugt et al. 2008) affects the distributions of found solutions for conceptual rainfall-runoff models.

The intuitive opinion on the model performance is frequently based on the hydrological criteria like the Nash-Sutcliffe coefficient (NSC) or the Persistence Index (PI). Although such indexes measure something else, both are maximized, both have the maximum in 1 and both suggest that model is “useful” if its value is positive (higher than 0). In this paper, it is found that the opinion on model performance based on the two criteria may be misleading, as similar largely positive values of NSC (≈0.7) observed on two different catchments may be accompanied by very contradictory PI values (≈0.6 at one catchment and highly negative at the other).