Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R

Highlights

• We answer ten basic questions on use of global sensitivity analysis on RR models.

• SA can be performed easily using the free Hydromad and Sensitivity packages in R.

• SA can help improve identifiability of a model by justifying fixing of parameters.

• With daily data a minimum duration for obtaining reliable SA results is five years.

• Simpler models have better-identified parameters but varying sensitivity.

Summary

Sensitivity analysis (SA) is generally recognized as a worthwhile step to diagnose and remedy difficulties in identifying model parameters, and indeed in discriminating between model structures. An analysis of papers in three journals indicates that SA is a standard omission in hydrological modeling exercises. We provide some answers to ten reasonably generic questions using the Morris and Sobol SA methods, including to what extent sensitivities are dependent on parameter ranges selected, length of data period, catchment response type, model structures assumed and climatic forcing. Results presented demonstrate the sensitivity of four target functions to parameter variations of four rainfall–runoff models of varying complexity (4–13 parameters). Daily rainfall, streamflow and pan evaporation data are used from four 10-year data sets and from five catchments in the Australian Capital Territory (ACT) region. Similar results are obtained using the Morris and Sobol methods. It is shown how modelers can easily identify parameters that are insensitive, and how they might improve identifiability. Using a more complex objective function, however, may not result in all parameters becoming sensitive. Crucially, the results of the SA can be influenced by the parameter ranges selected. The length of data period required to characterize the sensitivities assuredly is a minimum of five years. The results confirm that only the simpler models have well-identified parameters, but parameter sensitivities vary between catchments. Answering these ten questions in other case studies is relatively easy using freely available software with the Hydromad and Sensitivity packages in R.

Introduction

All too often the reporting of model evaluation exercises omits a key step, sensitivity analysis (SA), in helping to understand the performance of that model. Sensitivity analysis is the study of how the variation in the output of a model can be qualitatively or quantitatively apportioned to different sources of variation in the model input data, including model parameter values (Saltelli et al., 2000). It can be one of the simplest aids in diagnosing and remedying poor identifiability of models, to allow parameters to be more reliably estimated. However, as we illustrate below, when such analysis has been undertaken it is likely to have been a simple One-At-a-Time (OAT) sensitivity to parameters over some local range. Saltelli and Annoni (2010) have previously shown local/single point OAT analysis to be a perfunctory and inadequate method of sensitivity analysis unless the model is linear. If the model parameters are nonlinear, local or single point OAT cannot sample the space of parameters adequately. Hence they suggest using global SA methods, such as the Morris (1991) or the Sobol (1990) variance-based method. Because the Sobol method has a larger computational cost, they recommend the Morris method when model runs are constrained by computational time.

It is our impression that there has been a pervasive omission of SA in environmental model evaluation exercises, and that when SA has been invoked there has been an over-reliance on a local OAT approach. This is borne out by an illustrative analysis of rainfall–runoff modeling papers published in three journals in 2011: Journal of Hydrology, Ecological Modelling, and Environmental Modelling and Software. In 2011 these journals published 164 papers related to rainfall–runoff modeling (found when searching with the keywords: “rainfall–runoff” and “hydrological model” using Science Direct, http://www.sciencedirect.com). Of the 164 papers in that year, there are only 59 that include the words “sensitivity analysis” and only 11 of those papers (7% of 164 papers) that use SA methods for their rainfall–runoff modeling. Of the 11 papers, two papers use a variance-based method, one paper uses the Morris screening method, five papers use an OAT method and the other three papers use a heuristic, Bayesian or multi-parameter nonlinear SA method that employs neural networks as a pseudo simulator to reduce the computational burden of the analysis. The remaining 48 papers report either sensitivity analysis using a percentage change of input data to quantify the impact of climate change or just mention that they either did a sensitivity analysis or did not; that is they either considered scenario-type changes in inputs (impact analysis rather than model evaluation), or did not present any results on application of SA at all.

If OAT methods are removed from our count of papers, there are only 6 papers (4%) out of 164 papers that seem to use an adequate SA method to evaluate their rainfall–runoff models. It is clear therefore that model evaluation practices, as recently as 2011 for rainfall–runoff modeling, are demonstrably poor because even simple approaches of sensitivity analysis are not applied as a standard to investigate identifability. It is quite likely that such generally inadequate practices extend beyond rainfall–runoff modeling, not just in hydrology but in the environmental sciences more widely.

Of course there are exceptions to our general contention that sensitivity analyses of models reported in the literature tend to be perfunctory. For example, van Griensven et al. (2006) pointed out that over-parameterisation is a well-known problem in rainfall–runoff models with large numbers of parameters. They showed that adopting the Morris method with Latin Hypercube sampling can reduce the number of parameters in the Soil and Water Assessment Tool (SWAT) model for more efficient use. They also concluded that there are clearly different SA results between the catchments, and therefore each catchment model requires its own sensitivity analysis to select a parameter set to be used for model calibration or uncertainty analysis. To indicate another exception, van Werkhoven et al. (2009) undertook a Sobol-based sensitivity analysis of the Sacramento model in order to identify and fix insensitive parameters that have a total sensitivity index less than some threshold. While reducing the complexity of the calibration process via SA, the resultant model had essentially the same predictive performance to the non-fixed parameter case.

In this paper we show that an informative sensitivity analysis of a model need not require more than a few lines of code. Two SA methods, Morris (1991) and Sobol (1990), are used on four well-known rainfall–runoff models with four target functions, and four decadal daily datasets from five study catchments for illustrative purposes. For the analysis the SA package “Sensitivity” in R is used. It includes some global SA methods including the Morris and Sobol algorithms applied in this study. This package is linked to the “Hydrological Model Assessment and Development” (Hydromad) package (Andrews et al., 2011), an open source software package which is also built on the R platform. Hydromad and its user guide are available from http://hydromad.catchment.org. It includes the rainfall–runoff models investigated here except for the SIMHYD model which is called from Hydromad and also allows one to insert code for any conceptual hydrological model so that it can be evaluated.

The paper aims to show how sensitivity analysis can be easily used to help answer 10 important questions. All these questions are to some extent case-specific. While the method used is transferrable across case-studies, caution is usually required in transferring the results. In Section 2, we describe the two SA methods examined in this paper. Section 3 summarizes the catchments, input data, target functions and methods used for our analysis. In Section 4, we provide some answers to the 10 questions using the two SA methods. Section 5 presents a discussion of the study conclusions and our suggestions for future work. In Appendix A, we present a description of each of the four rainfall–runoff models used. Appendix B summarizes the hydro-climatic characteristics of the five catchments used in this study. In Appendix C, we present the simple code for implementing the two SA methods in R.