A model ensemble generator to explore structural uncertainty in karst systems with unmapped conduits

The study area is a 35 km² catchment located in the Northern Alps, on the border between Germany and Austria (Fig. 1a), with elevations ranging from 1,000 to 2,230 m above sea level (asl). The climate is cool, temperate, and humid, with a mean annual temperature of 5.7 °C and mean annual precipitation of 1,836 mm. Maximum rainfall occurs in June–August, while snow accumulates between November and May; Water Authority Vorarlberg data, 1961–1990).

Hydrogeologically, the catchment consists of a large karstified zone to the north, with primarily subsurface flowpaths, and a smaller, nonkarst zone of flysch to the south, generating surface runoff. The flysch zone consists of several sandstone, mudstone, and marl formations. Both zones drain to the Schwarzwasser Valley, which runs northeast and marks the boundary between karst and nonkarst (Fig. 1b). Two parallel flow systems run along the valley axis: a surface stream that collects water from the flysch zone, and an underground stream (i.e. a series of karst conduits) that collects water from the karst zone and infiltration from the surface stream. The two streams are connected by an estavelle (which is an orifice that may act either as an outlet or an inlet depending on groundwater conditions; QE), which demarcates the boundary between the upper and lower segments of the valley. The upper segment of the Schwarzwasser Stream receives surface inflow generated from the flysch zone and from a rockfall mass in the uppermost part of the valley, but no contribution from the karst zone. During low-flow conditions, the estavelle acts as a swallow hole, so that the streambed is dry until it receives new inflow from several downstream karst springs, described in the following. During high-flow conditions, the estavelle acts as a karst spring, so that the stream below this point includes a mixture of surface water and karst groundwater.

The karst aquifer developed in the pure, highly fractured and karstifiable Cretaceous Schrattenkalk limestone formation (approximately 100 m thick). The karst aquifer is highly permeable. The Schrattenkalk is underlain by the less permeable Drusberg marl (approximately 250 m thick) acting as an aquitard (Fig. 1c). The karst zone is heavily folded and fractured, resulting in strong structural geologic control on underground flowpaths. Major flowpaths in the karst aquifer follow the axes of plunging synclines, draining towards the southeast, before joining a series of conduits that parallel the Schwarzwasser stream flowing northeast (Goldscheider 2005). The aquifer discharges at three primary springs flowing into the Schwarzwasser stream: the estavelle (QE), described previously, at 1,120 m asl; Aubach Spring (QA), a large, intermittent overflow at 1,080 m asl; and Sägebach Spring (QS), the lowest permanent outlet of the system at 1,035 m asl (Table 1; Goldscheider 2005).

To test new modeling approaches, a well-accepted reference model is needed for comparison. Previous work by Chen and Goldscheider (2014) simulated the study catchment using a spatially distributed numerical conduit flow model: the Storm Water Management Model (SWMM; Rossman 2015). SWMM is described in more detail in section ‘Flow model: SWMM (EPA)’.

The conduit network map and the subcatchment boundaries used for the reference model were drawn based on extensive pre-existing geologic (Wagner 1950), speleological (Höhlenverein Sonthofen 2006), and hydrogeologic mapping, including numerous tracer tests (Goldscheider 2005; Goeppert and Goldscheider 2008). Flow inputs to the conduit model were generated by coupling it to a reservoir model representing recharge, storage and transfer of water in the epikarst and unsaturated zone, based on meteorological data from nearby weather stations. Spring discharge during the study period was recorded by four temporary monitoring stations (Chen and Goldscheider 2014). Extensive parameter estimation, sensitivity analyses, and calibration improved this model (Chen et al. 2017, 2018), and the final version is used in this study as a reference (Fig. 2).

For this study, the bulk of data used in previous modeling work was set aside to mimic the more common scenario of a minimally studied catchment. These data (tracer tests, cave maps, and approximate conduit maps) were reserved for future analysis and validation of the methodology. Datasets used in this study were limited to existing geologic and topographic maps, meteorological data, and spring discharge time series. Five overlapping geologic maps (including representative point strike and dip measurements, surface fracture traces, and representative cross-sections) were digitized and compiled in ArcMap (released by Environmental Systems Research Institute 2017) into one detailed map covering the entire catchment (Goldscheider 2005 and related maps; Chen et al. 2018; see the electronic supplementary material (ESM)) Topographic data were obtained as digital elevation model (DEM) raster files, with 50 m × 50 m pixels, from the State of Vorarlberg Digital Atlas. Meteorological data (hourly precipitation, air temperature, relative humidity) were recorded by Chen et al. (2018) from November 2013 to October 2014 at nine weather stations across the catchment, and were interpolated at a 100 × 100 m grid resolution. Discharge was recorded hourly for the same time period at the three major outlets of the system (QE, QA, QS; Chen et al. 2018).

Inflows to the karst system were calculated according to Chen et al. (2018). The meteorological data were fed to a lumped linear reservoir model representing water storage and drainage through the epikarst (Hartmann et al. 2012), paired with the HBV snow routine to account for snow accumulation and melt (Hock 1999). The output is a 100 × 100 m grid of fast and slow recharge time series across the entire catchment, which is then used as input for the groundwater flow model.

The first step in generating a structurally diverse model ensemble is to create a set of plausible three-dimensional (3D) geologic models within which the karst networks will evolve. In mountain catchments, it is generally accepted that the often-complex geologic setting can largely control the system’s flow behavior (Goldscheider 2011; Rogger et al. 2013), yet the exact shape of the contact surfaces between units is difficult to fully map, particularly in the subsurface. Representing the geologic structure of the site is therefore extremely important, particularly the shape of the lower boundary of the karstifiable unit, as this is where conduits often form.

The ensemble generator presented in this paper models the geology using an implicit approach (Lajaunie et al. 1997), with the open-source Python package GemPy (de la Varga et al. 2019; see ESM).

First, the stratigraphic sequence of the site was grouped into four major hydrogeologic units, in ascending order from oldest to youngest: a group of underlying low-permeability units, the low-permeability Drusberg marl, the Schrattenkalk limestone aquifer, and a low-permeability cover consisting of the Garschella sandstone and younger units (Fig. 1c). Next, point strike and dip orientation measurements for these units were located on the geologic map and the coordinates (X, Y, Z) were extracted from the DEM. Additional points were hand-picked from the map and from cross-sections to constrain the contact interfaces between units. Then, the model grid was defined based on the DEM, with a resolution of 251 columns × 200 rows × 52 layers, for a total of 2,610,400 cells. The horizontal cell length and width were taken from the 50 m × 50 m DEM, while the vertical cell height was manually selected to be approximately one-quarter of the thickness of the aquifer, or 27.5 m. The vertical resolution can be refined for more accuracy, but this significantly increases the computation time. The geologic model intentionally extends beyond the catchment boundaries to avoid edge effects, and was subsequently cropped.

GemPy interpolates the shape of the contact surfaces between units based on the stratigraphic sequence, and the orientation and interface data points. The resulting 3D model is exported as a GSLIB file recording which unit is present in each grid cell. To capture uncertainty with respect to the contacts, the elevation of the input data points could be perturbed, yielding multiple realizations of the geologic setting. However, this paper focuses on variations in the karst conduit network in a single realization of the geologic model (Fig. 3).

Generating plausible conduit network maps within the aquifer requires a computationally efficient karst evolution model. The ensemble generator in this study uses the Stochastic Karst Simulator (SKS), which models conduit evolution at a watershed scale, constrained by hydrogeologic knowledge, without solving the physical and chemical equations driving speleogenesis (Borghi et al. 2012). Conduit evolution is instead simulated based on the assumption that water will follow a minimum-effort path within the boundaries of the karstifiable units, computed using a fast-marching algorithm (Sethian 1996). This approach is sufficiently fast to allow the generation of many equiprobable, hydrogeologically plausible realizations (Borghi et al. 2012). This tool has been combined with parameter estimation to identify properties of the conduit network such as hydraulic conductivity of the matrix, number of major conduits, and conduit radius, for a small number of synthetic and real systems (Borghi et al. 2016, Sivelle et al. 2020).

The SKS includes three stochastic components: a discrete fracture network simulator (Borghi et al. 2015), an inlet point generator, and a randomization of the karstification impacting the hierarchy of the resulting cave conduit systems. Conduit network realizations within the same geologic setting differ from one another as a function of hydrogeologic parameters in SKS such as the ratio between matrix and fracture conductivity (Table 1). It is therefore possible to generate many different conduit networks for the same geologic setting. For this study, a subset of the available parameters was chosen to vary, and 100 SKS realizations were run, each using a different combination of parameter values selected randomly from a range (Fig. 4a,b). The parameter ranges were defined based on a combination of the recommended range provided in the SKS documentation and published fracture maps and orientations for the study site (Goldscheider 2002; Cramer 1959), but the ranges were also expanded to add variability to the models. The inlets to the system can be assigned as fixed locations (based on hydrogeologic observations such as known dolines or vertical shafts), random points within a defined spatial extent (if the actual inlet locations cannot be observed), or a combination of fixed and random locations (to account for both observed inlets and potential unknown inlets). For this site, lines of dolines and vertical shafts were observed along major synclinal axes. Based on these observations, in the reference model, every node along the branches of the network (which follow the synclinal axes) is treated as an inlet capturing runoff and infiltration from the area drained by that node, for a total of 20 inlet nodes. During the conduit network generation process for this study, to achieve a diverse ensemble of networks and avoid overly imprinting the SKS-generated networks with the reference network, the number of fixed inlet locations known from the reference model was varied in each run, but no random inlets were added. The runs with fewest inlets included only the fixed inlets corresponding to the endpoints of the reference network’s branches (“top5” in Table 2), while the runs with the most inlets included the uppermost 3 nodes of the reference network’s branches (“top13” in Table 2).

For each realization, SKS outputs a list of nodes and conduits making up the network, with one node in each model grid cell containing a conduit. This high-resolution network is then simplified by removing intermediate nodes along the conduits, using karstnet, a Python package for the statistical analysis of karst networks based on graph theory (Collon et al. 2017; Fig. 5; see ESM). The outlets of the system are then identified as the nodes located closest to the true (X, Y, Z) coordinates of the springs. To enable flow modeling, a new node is added at the true spring coordinates and a new conduit is created connecting the spring to the closest SKS-generated node.

Groundwater flow through each proposed conduit network was simulated using the Storm Water Management Model (SWMM) from the United States Environmental Protection Agency (Rossman 2015). SWMM has been used in several previous studies to simulate karst systems, with good results (Campbell and Sullivan 2002, Peterson and Wicks 2006, and Vuilleumier et al. 2019, among others), and performed well in previous efforts to model the site used in this study (Chen and Goldscheider 2014; Chen et al. 2017, 2018). Choosing the same flow model used in the reference model also enables easier comparison between model predictions from this study and from the reference model.

he SWMM is a dynamic rainfall-runoff and conduit flow model, that routes surface runoff into and through a series of linear subsurface conduit segments. However, it should be noted that SWMM is not designed to simulate contaminant transport, and it does not consider conduit–matrix exchanges, so it would not be suitable for systems with high conduit–matrix exchange rates (Peterson and Wicks 2006). For the Gottesacker karst system, although gradient inversion has been observed during high-flow events (Goldscheider 2005), the conduits generally drain the matrix. This is particularly true in the elevated parts of the system, where most of the karstified rock volume belongs to the unsaturated zone. In this setting, the conduits follow the troughs of the synclines, and lateral inflow can be observed from the “matrix” to the “conduits”, mostly with open-channel flow conditions in the conduits. This is different than the classical conceptual model of “conduit–matrix exchange”, where both zones are fully saturated, and the exchange can be described by means of hydraulic head difference between matrix and conduits and an empirical exchange coefficient.

For this study, although conventional conduit–matrix exchanges are not represented, inflow from the matrix to the conduits is inherently implemented in the model structure of SWMM, where subcatchments contribute flow towards a network of conduits. This approach is well-adapted to the hydraulic setting and karst aquifer configuration for this study site.

Flow through the conduits is represented by solving the Saint Venant equations (Eqs. 1 and 2), governing conservation of mass and momentum for the unsteady flow of water through a conduit network (Rossman 2015). When the conduit is unpressurized, Manning’s equation is used to compute the friction slope, but when the conduit becomes pressurized, the Darcy-Weisbach equation is used instead (Rossman 2015). The Saint Venant equations are solved in the flow routing model by converting them into an explicit set of finite difference formulas that compute the flow in each conduit and the head at each junction for each time step. The parameters required for each conduit are the length, diameter, elevations for entering and exiting conduits, and roughness.

Equations (1) and (2) represent Saint Venant equations for flow through a conduit, where x is the distance along the conduit (m), A is the cross-sectional area (m²), Q is the flow rate (m³/s), H is the hydraulic head of water in the conduit (m asl), S_f is the friction slope (calculated from Eqs. 3 or 4), h_L is the local energy loss per unit length, and g is the acceleration due to gravity (m/s²).

For this study, a Python module was developed to automate the process of converting SKS-generated conduit networks to a SWMM format, distributing recharge inflows across subcatchments within the system, assigning hydraulic parameters to the conduits, running the SWMM model, and retrieving and visualizing model outputs (see ESM).

For each SKS-generated conduit network in the ensemble, the node coordinates and the conduit connections between nodes are imported from text files to Python objects. Then, because the node locations are different for each network, the boundaries of the surface subcatchment drained by each node must be redrawn. This is done using a pre-existing Python package, pysheds, which computes the surface flow direction of each grid cell based on the DEM, and thus determines which grid cells drain to specified pour points (i.e., conduit network nodes; see ESM). The slow and fast distributed recharge timeseries data for each cell are then summed by subcatchment and converted to node “inflow” and “baseflow” timeseries (in the terminology of SWMM). Conceptually, this is a simplified representation of the epikarst collecting and funneling precipitation inputs into vertical shafts leading to the subsurface system.

Hydraulic parameters can then be assigned to each node and conduit in Python. The module developed for this study uses all of the preceding information to write a SWMM input file, runs SWMM, and then extracts the spring discharge timeseries at all three system outlets from the SWMM output file. All parameters and outputs for each run are stored as both Python objects and csv files. An additional function calculates basic statistics for each spring’s outflow timeseries: Nash-Sutcliffe efficiency, root mean square error, volume error, maximum, minimum, and mean discharge, and the percentage of time that each spring’s discharge is below a fixed threshold.

Full parameter optimization of all hydraulic parameters for each conduit network is too computationally expensive in the context of an ensemble modeling approach. Therefore, for this study, a minimalist parameter exploration strategy was adopted. All parameters except the conduit diameter were held constant, with values selected based on the upper and lower bounds used in the reference model, which themselves were selected based on common ranges for this type of karst system (Chen et al. 2018). A simple sensitivity analysis was first done on the reference model, to confirm that it was not highly sensitive to parameters other than conduit diameter. Diameter was found to be far more determinant of the spring discharge pattern than roughness, the only other parameter not already provided by SKS. The exact value of the conduit diameters was found to have far less importance than the location of conduit restrictions: conduits closer to the system outlets were more sensitive to restriction than conduits farther from the system outlets. Therefore, only two diameter values were considered: large (4.0 m) and small (1.0 m).

To limit the computation time for flow modeling in SWMM, a subset of ten conduit networks was first selected among the 100 SKS realizations, with the goal of representing the maximum diversity of possible networks. Diversity was based on how different each network was from the others, according to its outflow behavior.

To select the subensemble, each of the 100 networks was allocated four initial flow simulations, each with conduit diameter restrictions in different locations. To determine where to restrict the diameters, a hierarchical conduit order (similar to Horton ordering for surface streams) was calculated as the subcatchment area drained by that conduit over the total catchment area (Borghi et al. 2016). Since higher-order conduits, closer to the system outlets, are more sensitive to restrictions, the four initial flow simulations explore only the effect of restrictions in high-order conduits. For each network, the following four restriction scenarios were applied: (1) the highest-order 25% of conduits were restricted, (2) only the conduit immediately connected to QE was restricted, (3) only the conduit connected to QA was restricted, and (4) only the conduit connected to QS was restricted.

The worst-performing 10% of conduit networks were rejected based on the combined results of this first round of flow simulations. For each spring, the root mean square error (RMSE) was summed across all four initial flow simulations. Then, the conduit networks with the highest summed RMSE were dropped from the ensemble: 4% were dropped for QE, 3% for QA and QS (to prevent the behavior at any one spring from dominating the outcome), totaling a 10% rejection rate and resulting in a remaining ensemble of 90 conduit networks.

k-means clustering (Kriegel et al. 2017) was then used to select a reduced ensemble of the ten most-different networks, using spring discharge timeseries statistics as parameters: maximum, minimum, and mean discharge, and percentage of time that each spring’s discharge is below a fixed threshold (see ESM for a full explanation of k-means clustering methods).

For this study, each conduit network in the reduced ensemble of ten most-different SKS-generated networks was allocated an initial set of 20 SWMM runs, with different conduit diameter sets for each run. These 20 SWMM runs were then reduced to a subset of ten runs for analysis, first by rejecting the worst-performing 10% as nonbehavioral, then by selecting the ten runs with the lowest flow continuity errors, to form the final ensemble of 100 models (10 SKS networks × 10 SWMM runs).

All conduits were initially assigned large diameters (4.0 m), and a probability-weighted random set of conduits to restrict to 1.0 m diameter was sampled, without replacement, for each flow simulation. The probability of a conduit being selected for restriction was based proportionally on its order. The conduit diameters were sampled only from a binary distribution, and not from a Gaussian distribution. The goal of this strategy is not to optimize the conduit hydraulic parameters or to perform an in-depth analysis of the parameter uncertainty, but simply to represent a rough estimate of the uncertainty in spring discharge originating from parameter uncertainty, as opposed to uncertainty in the structure of the conduit network. Future work will explore alternative strategies for efficiently representing and minimizing parameter uncertainty, such as Monte Carlo Tree Search (Silver et al. 2016), or genetic algorithms (Karpouzos et al. 2001).

After rejecting nonbehavioral models and reducing the size of the ensemble, 100 models remain for analysis, each of which yields timeseries of predicted discharge at the three major springs in the karst system (Fig. 6). For comparison, the reference network was also allocated 20 SWMM runs with different parameter sets, using the same process as for the SKS-generated networks.

To enable further comparisons across the ensemble, a global error metric was calculated for each model, summarizing the overall goodness of fit between the model-predicted spring discharge and the observed spring discharge. The global error was calculated as the sum of the RMSE for each of the three springs, divided by the mean observed discharge at that spring (Eq. 3). This metric is only meaningful in terms of relative value, but it is useful to compare overall model fit across the ensemble.

Equation (3) shows the global error for each model in the ensemble. RMSE is the root mean square error at each of the three springs (QE: the estavelle, QA: Aubach Spring, QS: Sägebach Spring). Q͞obs is the mean observed discharge at each spring.

The geologic model generated by GemPy generally resembles the expected geologic structure as determined by previous field mapping (Figs. 1 and 3). The most apparent difference is that the upper contact between the Schrattenkalk limestone and the overlying Garschella sandstone, which in reality coincides with the land surface over much of the model extent, often appears several meters below the land surface in the GemPy model. However, the general shape of the Schrattenkalk unit, and particularly its lower boundary, resemble the expected structure as visualized in cross-section (Fig. 3c). Because conduits tend to form along the lower boundary of a karst unit, this boundary is more important than the upper boundary. The small vertical underestimation of the location of the Garschella-Schrattenkalk contact is therefore not concerning for the purposes of this study.

The initial ensemble of 100 SKS-generated conduit networks (Fig. 7) include networks that, subjectively, appear visually diverse and different from one another with respect to their overall configuration, their degree of branching, and their total number of conduits. However, almost none of these proposed networks visually resemble the reference network, such that they are not collectively exhaustive of conceptual space. The general structure of three or four branching conduits connecting almost perpendicularly to a main collector conduit along the axis of the Schwarzwasser Valley does exist in many of the SKS-generated networks, but the upper conduits rarely follow the synclinal fold axes as had been expected. Many of the proposed networks also bypass the Mahdtal Valley drainage axis (the northmost syncline) almost entirely, although there is clear and repeated tracer evidence for this connection (Goeppert and Goldscheider 2008). The only SKS network that closely resembles the reference network is No. 90 (Fig. 7, rightmost group of columns, row 90); however, this network was not selected by the clustering algorithm for inclusion in the subensemble.

Many networks that are visually dissimilar from each other performed equally well in predicting discharge timeseries with a good overall fit to the observed discharge. Additionally, many networks that were visually dissimilar from the reference network perform equally well or better (Figs. 7 and 8a).

The networks in the initial ensemble appear sensitive to structure but less sensitive to differences in parameter values. This can be observed visually in Fig. 7: rows where one of the SWMM runs in that row are highlighted as having a high global error (darker colors) generally include multiple high-error runs, rather than the high-error runs being scattered across different networks. However, even on networks where most of the parameter sets perform poorly, the parameter set where all the lower conduits are restricted results in much lower error (note the absence of any highlighted runs in the “lower” columns). These results suggest that for this initial ensemble, while the network structure contributes significantly to the differences in performance across models, some combinations of hydraulic parameters can also consistently affect model performance.

With only four parameter sets per network in the initial ensemble, a more quantitative sensitivity assessment would not be reliable; however, for the subensemble of ten SKS networks, with ten behavioral SWMM runs per network, a more detailed analysis is possible. The range of error values in the first round of four SWMM runs generally bracketed the range of error values in the ten runs on the subensemble (Fig. 8b). This validates the assumption that the first-round SWMM runs were representative of a wide enough range of behaviors to use in selecting the subensemble. The subensemble, though it does not include any networks that visually resemble the reference network, does indeed represent a diversity of means and variances in global error values, as well as appearing subjectively visually diverse in terms of general network configuration, number of conduits, and degree of branching (Fig. 8c).

Across the 100 models in the subensemble, plus the 10 uncalibrated SWMM runs on the reference network, the original, calibrated reference model had the lowest global error, as expected. However, all of the uncalibrated runs on the reference network had global error values within 15% of the reference model, suggesting that extensive calibration is not necessary to achieve an acceptable performance. Unexpectedly, for six out of the ten SKS-generated networks, at least one parameter set performed almost as well (global error values within 15%) as the calibrated reference model, and many performed as well or better than the uncalibrated SWMM runs on the reference network (Figs. 8b and 9). Even conduit network structures that are far from the reference (such as No. 91 or No. 86, which, respectively, lack connections known to exist from tracer tests, or include a large number of conduits that cross over structural anticlines to connect in straight east–west lines to the springs) were able, with minimal parameter selection, to yield passably good predictions of spring discharge behavior that were quite similar to predictions yielded by the same process on the reference network (Fig. 9). These structures are therefore not mutually exclusive in terms of flow behavior. Based on the performance of the SKS-generated conduit network that yielded the lowest-error prediction of spring discharge after testing only ten essentially randomly selected parameter sets, if that same network were subjected to further parameter optimization and calibration, it seems likely that it would be able to predict spring discharge behavior nearly as well as the reference model, despite having a very different structure. This suggests that the global spring discharge behavior is insufficient to discriminate between conduit network structures.

Returning to the model ensemble as a whole, there do not appear to be any detectable patterns governing which parameter sets or which networks perform better than others (Fig. 10). There do not appear to be categories of conduits more or less sensitive to restrictions, nor do there appear to be particular network configurations that yield higher or lower global errors. Certain networks are insensitive to parameter values, performing similarly under all conduit restriction conditions, while others are highly sensitive to parameter values. Some SWMM runs with restrictions in the lower conduits perform well, while other perform poorly, and vice-versa.

The only visible trend is that models with a higher percentage of restricted conduit segments tend to perform better (Fig. 11). This may be related to the binary nature of the diameter values used in this study, where conduits were assigned to be either wide or narrow, rather than being assigned diameter values selected from a distribution. Chen and Goldscheider (2014) indicate that the Gottesacker system also has variable conduit diameters along the principal drainage axis, and is uniquely sensitive to restrictions in these sections of the network, which may also contribute to the impact of conduit diameter in spring discharge predictions in this study. However, many models with few restricted conduits still perform as well as models with many restricted conduits. A high percentage of restricted conduits is therefore not a requirement for good performance. It should also be noted that Fig. 11 does not indicate which conduits are restricted. It is therefore possible that in the SKS-generated networks, spring discharge is only sensitive to restrictions in certain conduits, such that if only those conduits are restricted, the percentage of restricted conduits is low, but the model performance is good. Increasing the percentage of restricted conduits might then result in better performance only in that it increases the likelihood that the sensitive conduits are selected for restriction. However, even within this subset of more-sensitive conduits, there may not be a single unique combination of diameters that yields a best fit. Additionally, it is difficult to compare restriction locations across networks since conduit indexing is unique to each network. Because the conduit configuration is different in each network, parameter sets cannot easily be compared across networks, and the specific conduits that are most sensitive to restriction will be different in each network.

The lack of discernible patterns and the good performance of networks very different from the reference network support earlier findings by Borghi et al. (2016) that flow predictions are not sufficient to identify a unique network structure. However, although flow predictions are not discriminatory between network structures, they are more sensitive to the network structure than to the hydraulic parameters (in this study, the distribution of conduit diameters). This can be seen by comparing the mean distance between flow models within a network to the mean distance between flow models across networks. The distance between any pair of flow models (i, j) is calculated as the sum of squared differences (SSD) between the flow values predicted at each timestep (Eq. 4). The mean distance between models within a network is then the mean of the SSDs for every possible combination of pairs (n choose k) of flow runs within that network. The mean distance between models across different networks is the mean over all networks of the means of the SSDs for every possible pair of flow runs across every possible pair of networks (see ESM for a more complete explanation of SSD calculations). These calculations can be performed for each of the springs in the system, as well as for the sum of the SSDs at all three springs.

Equation 4 finds the distance between two models, calculated as the sum of squared differences (SSD), where Q_i(t) is the spring discharge predicted by model i at time t, and Qj(t) is the spring discharge predicted by model j at time t.

The mean distance between all flow runs on a single network can be thought of as the overall dissimilarity from one another of predictions generated by different parameter sets on that network, i.e., the importance of the parameters in determining spring discharge behavior. The mean distance between networks can be thought of as the importance of the network structure in determining spring discharge behavior. If the mean distance between networks is larger than the mean distances within each network, that would suggest that the network structure has a larger influence on spring discharge predictions than the parameter values. For the subensemble of networks considered in this study, the distance between networks is larger than the distance within networks in every case except for one network (No. 83), where the distance between models within the network is larger at the Sägebach spring only (Fig. 12). This suggests that, as expected based on previous work highlighting the impact of model structure on prediction uncertainty (Refsgaard et al. 2006; Enemark et al. 2019), the conduit network structure is more determinant of spring discharge behavior than the hydraulic parameters (in this case, the conduit diameter).

The ensemble approach to karst modeling demonstrated in this study provides a way to represent structural and conceptual uncertainty when the location of the conduits is unknown, by generating multiple competing hypotheses as to the network configuration. It is flexible in the choice of which conceptual aspects to vary, and how many computational resources to devote to parameter estimation. The bulk of the model code is open-source (with the exception of the 3D version of SKS; an open-source 2D version, pyKasso, is in progress (see ESM).

However, this approach can still be improved. For the flow models, epikarst routing is not considered, despite its importance for recharge processes and spring discharge curves. The groundwater flow processes modeled using SWMM are based on the assumption of perfectly circular conduit cross-sections, with only two possible diameter values: wide (4 m) and narrow (1 m). Conduit–matrix exchanges are not taken into account, limiting this approach to conduit-dominated systems. Additionally, the system inflow is subject to error because it is calculated based on automatically delineated subcatchments for each network, determined by a combination of surface topography and network node locations. This automatic process occasionally results in gaps where cells on the border between two subcatchments are not assigned to either catchment, and any precipitation falling on those cells is then not included in the system inflow.

For the conduit models, the conduit network subensemble used for the bulk of the analysis in this study is not exhaustive: it does not fully sample the conceptual space, as evidenced by the lack of any proposed networks resembling the reference network. This is due in part to the nature of the SKS conduit generation algorithm, which uses only the distance through the medium to the spring, computed using a fast-marching algorithm, and assuming a certain base level. Because it is assumed that water can flow anywhere when the reservoir is saturated, this distance neither accounts for gravity nor for real flow conditions, especially in the unsaturated zone. Finally, the network simplification (Fig. 5), which is necessary to reduce computation time, may oversimplify the conduit network configuration and obscure the original network proposed by SKS. Despite these conceptual gaps, many of the models in the ensemble were able to predict spring discharge behavior quite well.

The model ensemble generated for this study did not contain any conduit networks resembling the reference network. It was therefore not exhaustive. Many conduit networks in the ensemble that did not resemble each other or the reference network predicted the spring discharge behavior almost as well as the calibrated reference model, and as well or better than uncalibrated SWMM runs using the reference network. The networks in the ensemble were therefore also not mutually exclusive: spring discharge behavior was insufficient to discriminate between conduit network structures; however, the flow behavior is still useful as a low-cost first-pass filter to reject nonbehavioral models. Additionally, the conduit network structure did control model predictions of spring discharge behavior more than the flow parameters (i.e. the conduit radius).

These results support previous work indicating that, because model structure is a major contributor to prediction uncertainty (Bredehoeft 2005), approaches using multiple competing model structures are particularly desirable when a good understanding of prediction uncertainty is desired (Neuman and Wierenga 2003). It is not necessarily required to identify a single “best” network structure in order to benefit from a multi-model approach: simply acknowledging and incorporating structural uncertainty into the generation and interpretation of model predictions is useful in and of itself when making decisions based on those predictions.

Three main questions for further reflection arise from these results:

One of the initial goals of this study was to generate an ensemble of models that were collectively exhaustive, resulting in an ensemble that includes at least one structure that is close to the “true” structure. If this is not the case, as occurred for the model ensemble in this study, the ensemble can be narrowed down to a subset of more probable networks, and then repopulated using those networks as “seeds,” perhaps over the course of multiple iterations. In the initial ensemble of 100 in this study, only a single network (No. 90) resembled the reference network, and it was not selected as part of the subensemble of ten that was used for the bulk of the analysis. Two reasons for this appear possible: either the GemPy-SKS network generation process is lacking key information that went into creating the reference network, or the reference network is lacking information that is reflected in the networks proposed by SKS. The reference network is the result of the integration of large amounts of quantitative and qualitative information after many years of fieldwork and research on this particular karst system, so that although the precise location and properties of the conduits are uncertain, there is a very high degree of certainty that the general configuration of the network is correct. What, then, is missing from the GemPy-SKS network generation process, resulting in an ensemble with so little resemblance to the reference network?

The GemPy geologic model used as an input to SKS indicates only which formation is present in each model cell. It does not include any information as to the orientation of that formation in each cell, or the presence/absence of faults. Other data that was not included but is available includes tracer test results indicating known connections, speleological maps for known portions of the conduit network, and more qualitative information such as the presence of dolines aligned along synclinal fold axes in the upper part of the catchment. All of this information was used in the creation of the reference network. A first approach to integrating this information into the SKS-GemPy conduit network generation process could be to test adding each type of data individually, in order of increasing costliness of acquisition, regenerating an ensemble of networks at each step. This would allow for the identification of not only which types of data result in the most noticeable changes in the composition of the ensemble, but also which types of data provide the most information per unit cost. A first step could be to replace the SKS fast-marching algorithm with a flow simulation accounting for gravity, which is more computationally intensive and requires additional boundary condition information, but would provide more control over the conduit generation (Borghi et al. 2012).

Another goal of this study was to identify what additional data would be most useful in discriminating between models in the ensemble. Previous work by Borghi et al. (2016) suggests that solute transport predictions are significantly more discriminatory of conduit network structure than flow predictions. This is likely because solute transport is much more dependent on particle flow paths, whereas similar flow behaviors can result from many different flow paths. However, it is likely that multiple different structures will fit tracer data as well as flow data (Borghi et al. 2016; Sivelle et al. 2020). Although tracer test data will not identify a single “best” structure, integrating tracer test data will still reduce prediction uncertainty, particularly uncertainty in predictions of solute transport, by enabling the rejection of some structures in the ensemble. Tracer test results are therefore the logical next data to include if contaminant transport is a prediction of interest to the model end users. Because SWMM is not designed to model solute transport, simulating tracer tests on the networks in the existing ensemble would require translating them into a different modeling tool such as MODFLOW-CFPv2 (Reimann and Hill 2009; Reimann et al. 2013), which can be coupled to UMT3D to solve solute transport (Hu and Xu 2016), or FEFLOW (Trefry and Muffels 2007). However, a more qualitative use of existing tracer test data would be possible without running a complete transport simulation. Because much of the available tracer test data does not include continuous discharge measurements, the calculation of tracer recoveries involves some degree of uncertainty, but underground connections can clearly be identified and transit times, flow velocities and dispersion can be calculated. Many networks could be rejected by considering the networks as directed graphs and testing whether the points with connections demonstrated by tracer test data are also connected in those networks. Networks where demonstrated connections are missing, or where connections that have been demonstrated to be absent are present, would be rejected. Additionally, the available flow data could be considered more carefully. Rather than simply comparing the global fit across the entire available timeseries at all three springs, a more detailed analysis could focus on each individual spring, and on flow behavior under specific conditions (high, low, or intermediate flows). If the predictions of interest are the behavior of a specific spring under specific conditions, then such an analysis could be more discriminatory, without the need for any additional data collection.

The model ensemble simulations in this study, although they are based on minimal input data, can still provide useful information for practical water resource management applications in karst systems. The primary lessons to be learned are that: