Comparison of sensitivity analysis methods for pollutant degradation modelling: A case study from drinking water treatment
Highlights
► Five Sensitivity analysis methods are applied to a model of micropollutant degradation. ► Factors prioritisation, factors fixing and factors mapping are considered. ► Methods are compared in view of capturing non-linearity and factor interactions. ► Entropy-based methods are well suited in a risk assessment context.
Introduction
Sensitivity analysis (SA) is used to examine how a model output is influenced by the uncertainty of the model factors, where the term factors may include both model parameters and model inputs. In many cases it is unclear which sensitivity method to select a priori. The choice of a SA method depends on the objective of the specific study, the computational cost, the relationship between model output and model factors (linearity, additivity, monotonicity) (e.g. Saltelli et al., 2000, Saltelli et al., 2005, Yang, 2011). Whereas the computational cost of one simulation run can mostly be estimated at the outset of an analysis, it is often unclear how the uncertain model factors will interact and which simplifying assumptions will hold, thus permitting the use of simpler methods. Most SA applications within the scientific literature are based on changing one factor at a time (OAT) and assessing derivatives at a single reference point in factors space (local SA). Saltelli and Annoni (2010) have recently highlighted how this may lead to erroneous conclusions, especially when modelling environmental systems where non-linearity and factor interactions are abundant. For a case study from environmental technology the present study aims to address these points by comparing a wide range of SA methods considering the following objectives:
- •
Which factors are important in determining uncertainty about the model output?
- •
Which factors are non-influential in determining uncertainty about the model output?
- •
Which factors are responsible for producing model outputs in a specific region (e.g. failure region)?
Using the terminology of Saltelli et al. (2004) these three objectives are defined as follows:
- •
Factors prioritisation identifies the most important factors, i.e. the factors, which if known, would be expected to lead to the largest reduction of variance in the model output.
- •
Factors fixing identifies which factors can be fixed anywhere within their range of uncertainty without significantly affecting the variance of the model output.
- •
Factors mapping identifies which factors are responsible for leading to model outputs in a specified region, e.g. above a threshold value.
Saltelli et al. (2004) describe a fourth objective which is not further explored in this study: variance cutting identifies the minimal set of factors to be fixed to reduce the variance of the model output below a specified value.
This study applies five methods of sensitivity analysis, each of which is a representative technique from a broader class of methods. The selected methods are summarised in Table 1 and include: relative–relative sensitivity functions (based on derivatives, local SA), Morris Screening (screening method), Standardised Regression Coefficients (based on regression), Extended Fourier Amplitude Sensitivity Testing (variance decomposition) and the Kullback–Leibler divergence (based on entropy).
The SA methods are applied to a model that predicts the degradation of micropollutants for an ozone reactor in drinking water treatment. The findings of this paper can be generalised to any study of transformation processes in environmental technology when the objective is to predict the degradation or transformation of a compound or pollutant.
For the derivation of the model and the expert elicitation of uncertainty the reader is referred to Neumann et al. (2009), where the results from a sensitivity analysis obtained with the Extended-FAST method are discussed in detail from a process engineering perspective. The current study does not give a physical interpretation of the results but focuses on comparing five SA methods in view of three SA objectives.
The study is structured as follows: The theoretical framework for each of the five SA methods is introduced in detail, followed by a brief summary of the reactor model and the selected chemical compounds as well as a description of computational aspects and numerical settings. The results obtained with the five methods are presented, for the selected micropollutants. A discussion compares their performance in view of three SA objectives and assesses the broader implications for selecting an SA method when modelling pollutant transformation.
Section snippets
Framing and terminology
The objective is to apply and compare five SA methods to estimate how the predicted relative residual pollutant concentration Y (outlet concentration/inlet concentration) is affected by the uncertainty in the model factors xi. In this study the term factor or model factor is limited to describe model inputs and model parameters exhibiting epistemic uncertainty (uncertainty due to lack of knowledge).
Derivative-based methods: relative–relative sensitivity functions
The most frequently applied methods in SA are based on derivatives. They assess how the model
MC simulation
Fig. 1 shows the empirical cumulative distributions for the predicted relative residual concentrations Y for the three compounds (logarithmic scale). The mean relative residual concentrations Y for MTBE, bezafibrate and beta-cyclocitral are 0.71, 3.6e-2 and 3.0e-6 respectively. The cumulative distributions allow a probabilistic interpretation of the treatment performance: F(Y) on the ordinate is the probability of the relative residual concentration being below Y (abscissa).
Sensitivity analysis
The results obtained
Factors prioritisation
For factors prioritisation the effect of moving outside of the application ranges for both the derivative-based and regression-based methods is clearly observed when the non-linearity and interactions increase: The ratio of [sum(srr2):sum(β2):sum(S)] changes from MTBE [0.98:0.99:0.98] to bezafibrate [0.58:0.84:0.87] to beta-cyclocitral [0.001:0.25:0.50] where sum(S) identifies correctly the contribution of first order effects in explaining the variance of Y (, , ). Although the rankings
Conclusions
- •
Five sensitivity analysis methods based on derivatives, screening, regression, variance decomposition and entropy were applied to a model predicting micropollutant degradation in drinking water treatment.
- •
The results were compared with respect to factors prioritisation (detecting important factors), factors fixing (detecting non-influential factors) and factors mapping (detecting which factors are responsible for causing pollutant limit exceedances)
- •
The capability of the different methods in
References (21)
- et al.
An effective screening design for sensitivity analysis of large models
Environ Model Software
(2007) - et al.
Uncertainty in prediction of disinfection performance
Water Res
(2007) - et al.
Global sensitivity analysis for model-based prediction of oxidative micropollutant transformation during drinking water treatment
Water Res
(2009) Specification of design criteria based on probabilistic measures of design performance
Struct Saf
(2002)- et al.
How to avoid a perfunctory sensitivity analysis
Environ Model Software
(2010) - et al.
Modelling the fate of organic micropollutants in stormwater ponds
Sci Total Environ
(2011) Convergence and uncertainty analyses in Monte-Carlo based sensitivity analysis
Environ Model Software
(2011)- et al.
Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory
J Chem Phys
(1973) - et al.
An approach to the preliminary analysis of environmental systems
J Environ Manage
(1981) - et al.
A simple method of sample size calculation for linear and logistic regression
Stat Med
(1998)
Cited by (54)
Development of a sensitivity analysis framework for aquatic biogeochemical models using machine learning
2023, Ecological InformaticsPlant-wide assessment of alternative activated sludge configurations for biological nutrient removal under uncertain influent characteristics
2022, Science of the Total EnvironmentCitation Excerpt :The coefficient of determination (R2) obtained for all model outputs is above 0.9 (except for methane production in the UCT configuration where R2 is ~0.6) –see Table S2 in SI. High interaction indicates that model prediction is highly non-linear - i.e. the interaction between parameters is significantly high and the influence of influent parameters cannot be mainly captured by first-order effects (Neumann, 2012). The Si values are reported throughout the text inside parentheses.
The development of a calibration methodology for a realistic primary settling tank (PST) model
2021, Journal of Water Process EngineeringVariance-based sensitivity analysis: The quest for better estimators and designs between explorativity and economy
2021, Reliability Engineering and System SafetyImproved 1-D hydrodynamic simulation of reservoir release based on coarse resolution topography data using ISS-SRC-assisted ensemble Kalman filter
2020, Journal of HydrologyCitation Excerpt :Eqs. (27) and (19) indicate that both the EnKF and the SRC involve Monte Carlo realizations and utilize the subsequent covariance information, which implies that they are inherently connected. Therefore, instead of other GSA methods, the SRC (Neumann, 2012) is selected in this paper to describe the parameter-output relationship. Notably, the SRC formulation in Eq. (27) avoids the possible numerical instability caused by matrix inversion when parameters vary greatly in their magnitudes.