Evaluating the effectiveness of road mitigation measures

The first step is to identify the target species that prompted the mitigation and formulate the specific goals for mitigation. A list of target species is usually presented by the road authority responsible for the mitigation, often prepared in cooperation with other stakeholders such as wildlife managers and environmental planners. These lists may be based on (1) empirical studies, e.g., on road-kill or road-related changes in animal movements; (2) predictive (modeling) studies in which potential effects of mitigation measures are explored; and/or (3) expert-opinion. Occasionally, groups of species are targeted for mitigation, e.g., “small mammals”, “butterflies”, or “frogs”. This typically occurs as a result of expert opinion or when information is lacking. In such cases, a first step should be to specify targeted species to allow for an effective monitoring plan (van der Grift et al. 2009a).

Selection of target species for mitigation is based on considerations of human safety, animal welfare, and wildlife conservation. Human safety issues dominate when animal-vehicle collisions pose a significant risk to motorists. These species need not be of conservation concern. For example the construction of fences and wildlife crossing structures on Swedish highways is motivated primarily by concerns for human health risks associated with moose-vehicle collisions rather than a concern with the impacts of traffic mortality on the viability of moose populations (Seiler 2003). When animal welfare drives the selection of species, the motivation is that each animal affected by the road is one too many. For example, the death of high numbers of hedgehogs on Dutch roads is—although the survival probability of local populations may be affected (Huijser and Bergers 2000)—usually not believed to affect the existence of the species in the country, but if animal welfare is a concern, wildlife fences and crossing structures may be warranted. If wildlife conservation is the goal, target species for mitigation are selected on the basis of the potential impact of the road and traffic on species viability, e.g., determined through population modelling. This can include species with protected status as well as species of general conservation concern. Such species selection is generally directed by conservation legislation or environmental policies.

We distinguish two potential targets in road mitigation goals: (1) no net loss, and (2) limited net loss. No net loss implies that road impacts will be entirely mitigated, i.e., the post-mitigation situation for the targeted species and goals is identical to the pre-road construction situation. Limited net loss implies that a limited road impact will be accepted (van der Grift et al. 2009a). The target level should be decided in advance and will depend on the local situation. For example, in one jurisdiction a species may be common and its survival not significantly harmed by a small loss in cross-road movements, whereas somewhere else it may be essential to its survival, justifying a no net loss target. In case a limited net loss target level is selected, it should be carefully determined how much loss, relative to pre-road conditions, is acceptable. If this appears hard to pin-point, precautionary principles should be followed, i.e., no net loss should be selected as target level.

Currently, road mitigation studies rarely specify mitigation goals (see van der Ree et al. 2007). When goals are made explicit they are often too imprecise to allow for an evaluation of whether indeed they have been achieved, e.g., “allowing animal movement”, “restoring connectivity” and/or “promoting gene flow”. Effective evaluation of road mitigation measures requires a clear definition of success. We recommend the SMART-approach to develop goals that are Specific, Measurable, Achievable, Realistic and Time-framed (Doran 1981; examples in Table 1). The goals should ideally: specify what road impact(s) is/are addressed; quantify the reduction in road impact(s) aimed for; be agreed upon by all stakeholders; match available resources; and specify the time-span over which the reductions in road impact(s) have to be achieved. Well-described mitigation goals will channel the choices in the next steps towards an effective monitoring plan (Fig. 1).

The preferred approach is to study the responses of all species targeted for mitigation. However, the list of targeted species is often long and resources may not be sufficient to evaluate them all. It may also not be necessary to evaluate all targeted species, as some may be similar to others in their responses to roads and road mitigation. Hence, the species for evaluation need to be chosen, and a random sampling approach may be required if the general performance of a wildlife crossing structure needs to be assessed. However, studies of the performance of wildlife crossing structures usually target threatened species, i.e., species that are rare, in decline or both. If limited resources prevent an evaluation of all threatened species, one may choose species that are most likely to demonstrate statistically significant effects with comparatively little sampling effort in space and/or time. We suggest using the following criteria to select those species: (1) Select species that have shown strong responses to roads and traffic, e.g., species that are frequently killed by traffic, or species that have proven to be unable/reluctant to cross roads or who avoid the road corridor altogether (see also Jaeger et al. 2005). (2) Select species that are expected to show short response times to road mitigation, e.g., species that quickly habituate to wildlife crossing structures. (3) Select species that are relatively widespread, as this will increase opportunities to find suitable replication and control sites. (4) Select species with low natural variability in population densities over time, as high variability in population densities will decrease the statistical power to detect road mitigation effects. (5) Select species that can be readily and easily surveyed. If the list of selected species for evaluation, after applying these criteria, still exceeds available resources, further selections of species can be made on the basis of preferences, for example, even representation of different animal taxa, habitats and/or trophic levels.

As Table 2 shows there are many ways to measure road mitigation effectiveness, depending on the concern, i.e., human safety, animal welfare or wildlife conservation. The best measures, i.e., measurement endpoints, are those which are most closely related to the outcome(s) of real concern, i.e., the assessment endpoint (Suter 1990; Roedenbeck et al. 2007). For example, (changes in) population viability cannot be directly measured in the field, hence we measure attributes of the population that are known to be related to population viability and predict future likelihood of persistence.

Four measurement endpoints are suggested to assess effects of road mitigation measures on human casualties (Table 2). The best is the actual number of people killed or injured due to animal-vehicle collisions or due to collision avoidance. Alternative measurement endpoints include the amount of insurance money spent, number of hospitalizations due to animal-vehicle collisions or collision avoidance, or number of wildlife-vehicle collisions concerning species that potentially impact human safety, regardless of whether they resulted in human injury or death. Two measurement endpoints are suggested to assess effects of road mitigation measures on wildlife health and mortality, i.e., the number of animals killed or injured while crossing roads and the number of animals killed or with ill-health due to isolation from needed resources through the barrier effect of roads (Table 2). These measurement endpoints seem to complement each other as each endpoint addresses a different mechanism through which wildlife health and mortality can be positively affected by wildlife crossing structures, i.e., through a reduction in animal-vehicle collisions or through increased road permeability and hence increased access to resources. Therefore, we suggest to always use these endpoints together.

Eight measurement endpoints are suggested to assess effects of road mitigation measures on population viability (Table 2). The most informative measurement endpoint is the trend over time in the size (or density) of the local population. Trend in population size is fundamental to understanding how the species has responded to the road mitigation. For example, if existing roads are having population-level effects and crossing structures are successful in mitigating those effects we would expect to see increases in population size after the structures are installed. If the crossing structures are installed on a new road, successful mitigation would be indicated by no change in the size of the wildlife population. Population size itself is also related to population persistence, since smaller populations are more likely to go extinct by chance.

When it is not possible to estimate population size or trend, a reduction in road-kill numbers following mitigation may provide an indicator for mitigation effectiveness at population level, but only if compared with road-kill numbers at control sites (see also Step 4) and if assumed (which may not hold) that (1) mortality is the main mechanism through which roads affect the population, and (2) road-induced mortality is not counteracted by, e.g., increased reproduction or immigration. As both assumptions may not apply (but see Hels and Buchwald 2001), changes in road-kill numbers should be seen as less indicative than estimates of population size or trend. Similarly, reproductive success as an indicator for mitigation effectiveness at the population level should be used with care, as no increase in reproductive success following mitigation may be the result of higher reproduction levels pre-mitigation as a response to loss of individuals due to road mortality.

Age structure and sex-ratio within the population may provide additional information to supplement the estimate of trends in population size because roads may impact individuals of different age classes or sexes differently. A positive feature of these measurement endpoints is that changes may be detected sooner in population structure than in population trend. However, they are less closely tied to population viability so more extrapolation is necessary, and they are only applicable to species that show differential age or sex responses to the road or traffic.

Road permeability measurement endpoints, such as between-population movement and gene flow may also allow inferences to population-level mitigation, if the main population-level effect of the road is through movement (rather than, say, mortality). Increased movements between populations divided by roads may affect, e.g., dispersal success or access to mates (see, e.g., Mansergh and Scotts 1989) and consequently population dynamics. Migrations across wildlife crossing structures may restore gene flow and reduce road-related genetic differences between the populations (Gerlach and Musolf 2000; Vos et al. 2001; Epps and McCullough 2005; Arens et al. 2007; Björklund and Arrendal 2008; Balkenhol and Waits 2009; Corlatti et al. 2009). Although both measurement endpoints directly address the extent to which the barrier effect of roads is reduced, endpoint extrapolation is rather high because demographic and genetic connectivity between populations are not necessarily related to population viability.

An even less direct indicator of a change in population viability is a change in genetic variability within the population. Genetic variability is thought to be positively correlated with population viability (Frankham 1996, 2005; Lacy 1997; Reed and Frankham 2003; Reed et al. 2007). Small populations that result from increased mortality or habitat fragmentation lose genetic variability as a result of genetic drift or inbreeding (Keller and Largiader 2003). The disadvantage of genetic variability as an endpoint is that the correlation between genetic variability and population persistence is not well understood. However, changes in genetic diversity—as an important part of biodiversity—may in itself be considered as an assessment endpoint.

Appropriate study design, i.e., the spatial and temporal sampling scheme, is critical for determining the effectiveness of road mitigation. It is the responsibility of the ecologists involved in the research and monitoring process to ensure the design is rigorous and provides useful information. As argued by Roedenbeck et al. (2007), the optimal study design is a replicated BACI (Before–After–Control–Impact), where data are collected before and after road mitigation, both at sites where mitigation measures are being taken (impact sites—hereafter referred to as mitigation sites) and at sites that are similar to these sites but where no mitigation measures are taken (control sites).

Control sites are important to ensure that changes to the measurement endpoint can reasonably be attributed to the mitigation measures. For example, mitigation may reduce road-kill, but an observed reduction in road-kill could also be caused by other factors, such as a decrease in population density, increased road avoidance behavior or changes in traffic volume. An important assumption here is that mitigation and control sites are similar in all relevant respects (see also Step 6). As this assumption is rarely met, replication is strongly recommended for both the mitigation and control sites. We also recommend including unconventional controls or benchmarks that may further help to interpret observed changes, such as reference areas that are characterized by the absence of roads, or measurements of (national) trends in the selected measurement endpoints over time.

In some situations, there may be no suitable control sites available. Under these conditions, a replicated BA study design (Before–After) may be an alternative, where measurements are taken at multiple sites before and after the treatment. The fundamental limitation of this design is that an observed change in the measurement endpoint may have been caused by some factor other than the road mitigation. Since the BA design fails to distinguish other sources of temporal variability from effects of the mitigation measures, other potential impact factors (e.g., climate variability, increasing traffic volume over time) should be considered when interpreting the results (Roedenbeck et al. 2007).

In some other situations, such as when the effectiveness of an existing wildlife crossing structure is to be quantified, it may be impossible to collect any ‘before’ data. Under these conditions, a replicated CI study design (Control–Impact) may be possible, where measurements are taken at multiple mitigation and control sites after mitigation. The inference in a CI design is that differences between the mitigation and control sites are due to the mitigation measure. However, as no two sites are identical, this inference may be invalid if the observed effect arises from other systematic differences between control and impact sites, or possibly even random inter-site variation. Replication of both the mitigation and control sites increases the strength of the inference that observed differences are indeed due to the mitigation. Note that the level of replication required for a CI study is higher than the level of replication required for a BACI type of study.

When selecting an appropriate study design, opportunities for experimental manipulations should be explored, as this may provide higher inferential strength. For example, if the construction of wildlife crossing structures along one road can be staged, the temporarily non-mitigated stretch can be used as control site. Or when a time gap can be planned between the construction of wildlife fences and the construction of crossing structures or some of the crossing structures can be closed intentionally for a while after construction, the individual effects of both measures can be evaluated. Such an approach requires that goals and plans for evaluations are incorporated into the construction schedule.

Several key questions related to data collection should now be addressed: (1) How long should sites be monitored before and after road mitigation? (2) How often should sites be monitored? (3) How many replicates are needed? As these decisions are unlikely to be independent, we recommend conducting model-based power analyses to optimize the sampling design (see, e.g., van der Grift et al. 2009b). For example, Fig. 2 illustrates the relationship between mitigation effectiveness (the expected effect size) on the degree of temporal replication needed for adequate statistical power. Similar graphs can be produced for other design variables such as sampling frequency and the number of replicate sites. Note that either pilot studies or pre-existing data on anticipated effect sizes are needed to conduct this type of analysis.

The sampling scheme is related to the chosen measurement endpoint and the characteristics of the studied species. For example, for a highly mobile species with a long lifespan, monitoring over a longer period would be required to assess a change in population density than that required to detect a change in movement. Similarly, a shorter monitoring period would be required to assess a change in road-kill numbers for a species that crosses roads frequently than for a species that crosses roads infrequently. For some measurement endpoints, such as changes in population size/density, higher levels of replication will allow a quicker evaluation of effectiveness. A study with three replicates will need to be continued for longer than a study with ten replicates, because with more replication the uncertainty in effect size will be reduced, thus allowing a reliable decision to be reached sooner.

The rate of use of wildlife crossing structures often increases over time (e.g., Clevenger and Waltho 2003; Ford et al. 2010) due to habituation or gradual improvement in the quality of the crossing structure (e.g., vegetation succession on wildlife overpasses). If usage and effectiveness are positively correlated, it is also likely that effectiveness will increase with time since mitigation. Therefore, if monitoring ceases too quickly, an incorrect inference that a crossing structure is ineffective may be drawn. In fact, in some cases monitoring resources may be more effectively allocated by waiting for a few years after installation of the mitigation measure before starting the ‘after’ monitoring. This may be particularly true when the assessment endpoint is population viability. Similarly, monitoring a site for too long commits resources after they are needed. Thus, sampling should not begin before an effect is expected to have occurred and should continue long enough to detect lagged and/or transient effects. A worst-case scenario is that the sampling duration is too short to detect a real effect and that future mitigation projects reject the use of a measure that is, in fact, successful.

Sampling should not just be limited to the selected measurement endpoint. Other variables should also be measured to improve interpretation of the results, provide better comparisons among study sites, and allow for stronger inferences concerning the causes of observed differences. We recommend documenting spatial (among sites, where appropriate) and/or temporal (within sites over time, where appropriate) variability in: (1) road design and traffic, (2) crossing structure design and use, and (3) structural features of the surrounding landscape, all of which have been shown to influence the use of road mitigation measures (Clevenger and Waltho 2000; McDonald and St-Clair 2004; Ng et al. 2004; Clevenger and Waltho 2005; van Vuurde and van der Grift 2005; Ascensão and Mira 2007; Grilo et al. 2008).

Road design covariates should include road width, road surface elevation (elevated road bed or road bed in cut), presence and type of pavement, presence and type of street lights, presence and type of fences, presence and type of noise screens, presence and width of median strip, presence and type of barriers in the median strip, presence and width of road verges, and presence and type of vegetation in road verges. Traffic volume and speed should be documented at several temporal scales (e.g., daily, seasonally, annually). Road mitigation covariates should include size and characteristics of the crossing structures, the type and size of wildlife fences, passage use by the target species and non-target species, and presence and frequency of use by humans and domestic animals. Information on the duration of the construction period that marks the transition from the ‘before’ to the ‘after’ situation and the date that road mitigation measures were ready for use may also be important. Finally, landscape covariates should include the altitude, topography, land use, type and amount of vegetation and the occurrence of characteristic landscape elements (e.g., river, pond, hedge row).

For most measurement endpoints, several survey methods exist (Table 3) but not all methods are equally effective for all species or species groups. We recommend survey methods that monitor multiple species simultaneously to provide more information for similar effort. We also recommend using more than one survey method for each species, because combining methods can decrease bias and provide better estimates of the variable of interest. Consistent use of the same methods and personnel over time and across control/mitigation sites is important to provide comparable results.

A comprehensive evaluation of road mitigation measures will require a substantial budget. However, other resources that may not have direct costs are equally important, e.g., sufficient time, or stakeholder support. The need for both economic and non-economic resources demands detailed organization and planning, including clear deadlines for decisions, and strong consensus among the research team, the funding organization and other stakeholders. For example, if a land owner refuses access to a sampling site during a long-term study, resources spent on sampling that location will have been wasted.

If all necessary resources cannot be guaranteed before commencing the project, carrying out the study should be seriously reconsidered. Hence, we recommend conducting a comprehensive risk assessment before the start of a study, to judge its feasibility. Such risk assessment should not only address costs, but all types of resources needed for the study, including risks related to the research itself. As road mitigation evaluation studies are ambitious, especially those that aim for measuring effects on population viability, unexpected complications are likely to arise and thus uncertainties should be incorporated into cost and scheduling estimates. For example, a selected study site may become unsuitable during the study due to changes in land use, or a positive trend in population size due to road mitigation may be observed but more years of measurement than planned seem to be needed to provide statistically significant results. Preferably, the monitoring plan includes an analysis of such risks and presents practical solutions on how to avoid them and what to do when they are unavoidable. For example, if our sampling scheme is based on ten replicates, we may select and sample at two more sites (i.e., for a total of 12), as a back-up for sites that may unexpectedly become unsuitable during the study.

The feasibility of a study can be easily increased using the protocol described in this paper, as at most steps there is choice on how to proceed (Fig. 1). Hence, when one or more resources are expected to be limiting, a different decision at one or more steps (e.g., choice of target species or measurement endpoint) may provide a practical solution. We do not recommend, however, leaving out essential components of an evaluation, such as the measurement of covariates, an often underestimated part of evaluation studies in terms of effort and budget, as this will considerably reduce the inferential strength of the study, limit the possibilities to compare study sites or extrapolate, and decrease the ability to explain the results. Hence, if choices have to be made, we recommend conducting one scientifically rigorous study that is more likely to contribute new knowledge than numerous poorly-designed studies.