Effectiveness and implications of spatial background restrictions on model performance and predictions: a special reference for Rattus species

Some rodent species in the Philippines pose significant economic and health challenges. Certain Rattus species, for example, negatively impact rice yields (Aplin et al. 2003), act as vectors for human pathogens (e.g., Mendoza and Rivera 2019), and affect populations of endemic rodent species as invasive competitors (Rickart et al. 2011), making their management a national priority. Therefore, generating accurate statistical habitat-based analyses for these species may contribute to optimizing rodent management strategies in the country. However, due to the limited availability of geographical records for many rodents, particularly within the Rattus species in the country, this study focuses on those Rattus species for which there are sufficient occurrence records to construct a model (see Hernandez et al. 2006; Pearson et al. 2007; Proosdij et al. 2016; Sampaio and Cavalcante 2023). Thus, the study only includes three invasive species: Rattus tanezumi (Oriental Asian House Rat), Rattus norvegicus (Brown Rat), Rattus exulans (Polynesian Rat), and one endemic species, Rattus everetti (Philippine Forest Rat). We collected occurrence data of the four Rattus species from various sources, including the Global Biodiversity Information Facility (GBIF) (https://​www.​gbif.​org), the synopsis of the Philippine Mammals (https://​www.​fieldmuseum.​org), and data records from the University of the Philippines Los Banos-Museum of Natural History (UPLB-MNH). The records acquired from the two latter sources were georeferenced following the recommendation of Wieczorek et al. (2004). Meanwhile, to ensure record data accuracy sourced from GBIF, we used the package “CoordinateCleaner ver 3.0” (Zizka et al. 2019) in R ver. 4.1.2 (R Core Team) to help us eliminate data associated with country or province centroids, open oceans, and locations of biodiversity institutions such as museums, zoos, and universities, as well as to check for and remove any duplicated records. This resulted in a dataset comprising 201 records for R. everetti, 170 for R. tanezumi, 157 for R. exulans, and 23 for R. norvegicus (see Supplemental Fig. S1).

For our environmental data, we utilized a 10-m resolution land cover map with ten categories (ESRI 2021), nineteen climatic variables at 30 arc-seconds (~ 1 km by 1 km spatial resolution, www.​worldclim.​org) sourced from Fick and Hijmans (2017). Further, we also used ~ 1 km by 1 km resolution data from Venter et al. (2018) that represents cumulative human pressure on the environment. This dataset quantifies the impact of human activities like settlement, agriculture, and transportation on natural landscapes, essential for assessing the impact of habitat degradation on the occurrence of species. We employed the R package “multiColl ver 2.0” to assess cross-correlation among the thirty environmental variable predictors and reduce the risk of incorrect inferences and predictions (Salmerón et al. 2021). We applied a cut-off threshold to mitigate the influence of highly correlated variables. Criteria such as pairwise correlation of coefficients |r| exceeding 0.7, Condition Numbers (CN) surpassing 30, Determination of Correlation Matric (D) approaching 1, indicating no collinearity, and Variance Inflation Factors (VIF) exceeding ten guided the selection of robust environmental predictors for Rattus species (Dormann et al. 2012). All these data were processed within GRASS GIS to aggregate them into a ~ 1 km by 1 km spatial resolution dataset (see Supplemental Tables S1 and S2).

The data records for four Rattus species may lack uniformity across their distribution ranges, as some regions were more heavily sampled than others (Supplemental Figs. S1 and S4). This selective sampling can lead to uneven distribution of records across the country. To address these sampling biases, we guided the background data selection by implementing various spatial background restrictions, detailed in the subsequent sections.

According to Merow et al. (2013), users have considerable flexibility in determining how to distribute background points within the study area. This flexibility is crucial because mitigating sampling bias often lacks universal guidelines or solutions (Fourcade et al. 2014). This is particularly true when dealing with data from archival sources, where we frequently lack prior knowledge about the inherent biases in the datasets. Thus, we adopted four key approaches to restrict background selection across the study area for predicting habitat suitability for Rattus species. First, we only restricted the background data selection to the islands with only documented species occurrence records (Supplemental Fig. S1). This involved excluding islands that lacked documented occurrence records based on the available records (Supplemental Fig. S5). Secondly, we referenced the International Union for Conservation of Nature (IUCN) to delineate the extent of occurrence for each Rattus species (concept adapted from Ranc et al. 2017; Whitford et al. 2024). The range maps provided by IUCN, represent the extent of occurrence for Rattus species (see Supplemental Fig. S2), guided our exclusion of islands without the historical presence of these species, ensuring the background data selection was confined to areas within their verified range (Supplemental Fig. S6). The third approach utilized a road density map of the study area sourced from the National Mapping and Resource Information Authority (NAMRIA) (https://​isportal.​namria.​gov.​ph), focusing on distances from roads (Kadmon et al. 2004). We restricted the background data selection within a 5 km distance from roads, based on computational analysis showing that approximately 90% of four Rattus species records fell within this range (Supplemental Fig. S3). The decision to restrict background data selection to within a 5 km radius of the road arises from practical limitations in field data collection. This radius is presumed to represent the maximum distance within which data collectors can feasibly access and place traps, as areas beyond this range are considered less accessible to observers (Supplemental Fig. S7). Lastly, we defined three buffer width areas from the occurrence points (e.g., Whitford et al. 2024) with radii of 20 km, 40 km, and 80 km to restrict the background area selections, based on the premise that these areas correspond to varying levels of surveyor access, which in turn influences data collection for Rattus species. Each buffer width area represents a hypothesized degree of accessibility, critical for modeling the potential distribution of data collection efforts across various geographic extents. This approach allows us to systematically assess how differences in access levels impact the performance of our models for Rattus species. The 20 km radius signifies areas that are highly accessible, facilitating frequent and comprehensive data collection (Supplemental Fig. S10). The 40 km radius encompasses moderately accessible areas, often more remote, where data collection can be less consistent (Supplemental Fig. S11). The largest buffer, with a radius of 80 km, represents the outermost practical limit for surveyor operations, where data collection is expected to be the least frequent, largely due to logistical constraints to cover this extensive range (Supplemental Fig. S12). 

Furthermore, we implemented a combined approach by integrating a 5 km road distance with other three spatial background restrictions: islands with documented occurrences, IUCN ranges, and buffer width areas. This methodological integration aims to evaluate whether the inclusion of road features enhances the utility and effectiveness of three spatial background restrictions, compared to using these restrictions independently. Firstly, we used combined spatial background restrictions of the island, which had documented species occurrence records along with a 5 km road distance (Supplemental Fig. S8). This approach allows us to restrict the background area to islands with documented species occurrences, while also considering road accessibility within these islands. Secondly, we employed a method that combined spatial background restrictions based on the IUCN species’ extent of occurrence and 5 km road distance (Supplemental Fig. S9). This approach was designed to enable the model to incorporate information from both ecologically significant areas for Rattus species and areas that are accessible for data collection by observers, all within the species’ extent of occurrence as determined by the IUCN. Lastly, we also integrated the 5 km road distance within buffer areas of varying widths (20 km, 40 km, and 80 km), this combined approach aims to assess the differential impacts of road distance on the coverage of potential survey areas across a range of buffer distances in improving the model performance of Rattus sp. (Supplemental Figs. S13 to S15).

In this study, we compared and evaluated two different methods for distributing background sampling points using the FLEXSDM package version 1.3.4 (see Velazco et al. 2022). The first method, as widely documented (Phillips and Dudík 2008), involved randomly distributing the background sampling points across the entire study area to ensure a balanced environmental representation (Phillips et al. 2006). Our second method utilizes a biased distribution of background sampling points, concentrating on or giving more weight to areas of high sampling intensity (e.g., Ranc et al. 2017) for each Rattus species. We generated species-specific density maps using a Gaussian Kernel (Elith and Leathwick 2009; Elith et al. 2011) and utilized these maps as a bias file to influence the distribution of background points. This method assumed that high-density areas on the maps (refer to Supplemental Fig. S4) indicate regions of heightened historical sampling effort. Background points were, therefore, sampled more frequently from these probability surfaces, increasing the likelihood of selecting areas with more extensive sampling histories (Barber et al. 2021). This strategy is designed to enhance the model’s capacity to highlight specific habitats occupied by the species, rather than just areas that are heavily sampled (Ferrier et al. 2002; Dudik et al. 2005; Phillips et al. 2009). Refer to Supplemental Fig. S5 to S16 for a more comprehensive illustration.

Given the extensive range of our study area, we used 10 000 background sampling points to contrast with the presence location records for each of the four Rattus species, following the recommendations of Phillips and Dudik (2008) and Barbet-Massin et al. (2012). This number of points is recognized as sufficient for building predictive models, effectively enhancing overall model performance. Using random and biased distribution methods, we analyzed the spatial background restrictions separately, creating distinct model scenarios (refer to Table 1). In our analysis, we identified two of the twenty-four model scenarios as “reference model” that did not incorporate the specific spatial background restrictions developed for this study (Supplemental Fig. S16). The first reference model employed a random method, distributing background sampling points uniformly across the study area. The second reference model used a biased approach, utilizing a density map as a bias file to strategically distribute background sampling points. These reference models established baselines for assessing the impact on model performance when spatial restrictions are applied, compared to scenarios where the background area is unrestricted by geographical information.

We used the Maximum Entropy (MaxEnt) modeling approach to demonstrate the objectives of our study, specifically utilizing the `SDMTune` package ver. 1.3.1 (see Vignali et al. 2020) to build the final models for Rattus species. We used `MIAmaxent ver.1.2` (Vollering et al. 2019), which aided in selecting important environmental variable predictors for each Rattus species (see Supplemental Table 3). This package provides tools for user-controlled transformation of explanatory variables, selection of variables by nested model comparison, and flexible model evaluation. This method ensures simpler yet effective models with reduced overfitting risk, focusing on variables that significantly enhance predictive accuracy for each Rattus species (Vollering et al. 2019). For further demonstration, we computed the variable importance (%) of the final environmental variables for each Rattus species using `SDMTune` package ver. 1.3.1 (Supplemental Figs. 17 to 20). Additionally, we generated the marginal response curves for each model scenario with respect to these variables (Supplemental Figs. 21 to 28). We also illustrated the response curves under two different methods of distributing background points: the random method and the biased method (Supplemental Figs. 29 to 31).

We partitioned the dataset using a threefold cross-validation approach to optimize it for both training and validation purposes. Additionally, to assess the robustness of our models, we carried out 100 individual runs for each species model. We then averaged the outcomes of the 100 replications to create a single predictive map for each species per model scenario. A single map represents the average predictive metric outcome across all 100 iterations, providing a more reliable result of the model’s performance. We evaluated the model’s accuracy using two key metrics: the Area Under the Curve (AUC) and True Skill Statistics (TSS). A value greater than 0.5 for the AUC metric suggests that our model’s prediction is better than chance (Liu et al. 2018). For the TSS metric, a value below 0.40 indicates poor model performance, while a value above 0.8 is considered excellent (Pramanik et al. 2018).

Furthermore, we assessed habitat suitability for each species generated from different model scenarios on a scale from 0 to 1.0. Scores between 0 and 0.4 imply very low suitability, scores between 0.6 and 0.8 indicate moderate suitability, and scores above 0.8 up to 1.0 signify very high suitability (Pramanik et al. 2018). Additionally, we employed Schoener’s D as a metric to quantify and assess the extent to which model scenarios employing spatial background restrictions produced habitat suitability patterns similar to those of the reference model. Values of Schoener’s D approaching 1 signify a high degree or complete similarity, whereas 0 denotes no similarity (Schoener 1968). To facilitate the interpretation of these similarities, we adapted the categorical ranges suggested by Rödder and Engler (2011) and Santamarina et al. (2023). Specifically, we defined the ranges in this study as follows: 0–0.5 indicates no or very low similarities, 0.5–0.7 denotes low similarities, 0.7–0.8 represents moderate similarities, 0.8- 0.9 indicates high similarities and 0.9–1.0 signifies very high similarities.

In evaluating twenty-four model scenarios across four Rattus species, the random background distribution method yielded higher performing models than the biased background distribution method across Rattus species (Fig. 1). Specifically, for the three species—R. everetti, R. exulans, and R. tanezumi—each with over 100 recorded occurrences (Fig. 1a to f), showed a consistent pattern of high performance in models S3 (random background points + 5 km road distance), S4 (random background points + 5 km road distance + documented known occurrence), and S5 (random background points + 5 km road distance + IUCN species occurrence range). The models demonstrated good performance, achieving AUC values between 0.85 and 0.88 while the TSS values ranged from 0.56 to 0.62, further confirming the model’s reliability (Fig. 1a to f). For R. norvegicus, which had the fewest records at 23, the highest model performance was observed under models S1 (random background points + documented known occurrence), S2 (random background points + IUCN species occurrence range), and S12 (reference model under random method). Each model achieved AUC values of 0.85 to 0.86, while the TSS values ranged from 0.59 to 0.60 indicating good performance (Fig. 1g, h). Across all four species, restricting the spatial background to a 20 km buffer size, whether using a random or biased distribution of background points, consistently resulted in lower model performance across twenty-four different scenarios.

Full size image

Our analysis showed that some model scenarios with spatial background restrictions had high to very high similarities with reference models S12 (random) and S24 (biased) based on Schoener’s D index, as demonstrated in Figs. 2, 3 and Supplemental Figs. 32 to 34). Within the random background sampling point distribution, models S1, S2 and S8 demonstrated high similarities (D > 0.9) in habitat suitability with the reference model S12 for R. everetti (see Figs. 2 and 3). For R. tanezumi, models S1, S2 and S3 showed high similarities (D > 0.9) with the reference model S12 (Fig. 2 and Supplemental Fig. S32). In the case of R. exulans, models S1, S2, S3, S4, S5, and S8 exhibited high similarities (D > 0.9) with the reference model (Fig. 2 and Supplemental Fig. S33). In terms of model scenarios that surpassed the AUC of the reference model (S12), our analysis showed that for these three Rattus sp., restricting the background area to a 5 km road distance alone (S3) or in combination with other spatial background restrictions (S4, S5, S10, and S11) performed better than the reference model (S12) (Fig. 1). This demonstrates their enhanced capability to improve model performance and address sampling bias in the available datasets of these three Rattus species, compared to simply distributing the background sampling points randomly across the study area (refer to Fig. 2). For R. norvegicus, which has only 23 occurrence records, all model scenarios except model S6, S7, S9, and S10 demonstrated high to very high similarities in habitat suitability predictions compared to the reference model (see Fig. 2 and Supplemental Fig. S34). However, none of these model scenarios outperformed the reference model (S12) for this species (see Fig. 2). Meanwhile, model S6, which utilizes a 20 km buffer width size, consistently recorded the lowest AUC values across all four Rattus species, indicating its negative impact on model performance. Additionally, integrating road features within the 20 km buffer did not notably improve model performance, especially noticeable in R. norvegicus (refer to Fig. 2).

Full size image

Full size image

Employing the biased background sampling point distribution method yielded varied results across different Rattus species. For R. everetti and R. tanezumi model S13, which focused on islands with documented occurrences, S14 utilizing the IUCN species range, and S19 and S20 employing 40 km and 80 km buffer sizes, respectively, demonstrated a very high degree of similarity (D > 0.95) in habitat suitability to the reference model (S24) (Figs. 2 to 3 and Supplemental Figs. 14 to 15). As for R. exulans, all model scenarios exhibited high similarities (D > 0.9) with the reference model. Notably, spatial background restrictions incorporating a 5 km road distance alone (S15) or with other spatial restrictions (S16, S17, S22, and S23) consistently outperformed the reference model (S24) for these three Rattus species (Fig. 2). In the case of R. norvegicus, all model scenarios showed high degrees of similarity (D > 0.9) with the reference model (S24) (Fig. 2 and Supplemental Fig. S16). However, none of these model scenarios surpassed the AUC performance of the reference model, with all closely matching its performance for R. norvegicus predictions (Fig. 2). Meanwhile, model scenarios implementing a spatial background restriction of a 20 km buffer size consistently showed the lowest AUC values across all four Rattus species, highlighting its negative impact on model performance even with a biased distribution method (Fig. 2).

Our analysis confirmed that the variable importance of key environmental factors, particularly BIO1 (Annual Mean Temperature), maintained consistent significance across all model scenarios for each of the four Rattus species studied. This uniformity is evident from the highly stable marginal response curves of BIO1, which showed no to minimal variation across different scenarios employing various methods of spatially restricting the background area. This consistent behavior highlights BIO1’s strong predictive power and establishes it as a critical determinant in predictive models for Rattus species. Conversely, other environmental variables displayed significant variability in their marginal response curves across scenarios. This inconsistency indicates that these less influential variables are more susceptible to fluctuations in model settings and background conditions, thereby affecting their reliability and influence on the predictive outcomes (Supplemental Figs. 21 to 28). This pattern of consistency and variability extends to scenarios employing both random and biased methods for distributing background sampling points (Supplemental Figs. 29 to 32).

This study has a key limitation: our analysis focused exclusively on geographical information. Despite this, we have still demonstrated the importance of applying spatial restrictions to background data in presence-only models, especially for Rattus species. It shows that specific methods can significantly enhance model performance, providing an initial strategy to mitigate sampling biases for Rattus species. For instance, spatially restricting background selections within a 5 km road distance has been shown to enhance model performance for predicting Rattus species. However, the choice of buffer width size is critical; a spatial extent that is too narrow, such as 20 km buffer width used in this study, failed to capture the species’ broad ecological variability, potentially negatively impacting model performance. The selection of background distribution methods for species distribution modeling, whether random or biased, should be informed by their effectiveness in accurately representing the environmental conditions crucial for the species under study. Each method has unique advantages that should align with the spatial distribution of the available data. A random distribution method is typically preferred by many because it captures a broad ecological spectrum and prevents bias towards areas with dense data accumulation. This approach ensures that the model comprehensively represents diverse environmental conditions, essential for accurately generalizing species distribution across varied habitats. Conversely, a biased distribution method is more likely advantageous in regions where data are densely clustered. By focusing the background data in these areas, it corrects for potential over-representation and achieves a balanced environmental portrayal across the model’s entire range. This strategy is crucial for maintaining the accuracy of the model, especially in studies where specific locations are disproportionately sampled. Ultimately, the choice of distribution method must be tailored to the specific needs of the study, ensuring that the model reflects the true environmental conditions necessary for reliable and accurate predictions. Furthermore, an important observation from this study is that while mitigating sampling bias is crucial, the quantity of species occurrence records is even more critical. This is particularly significant when the background area selection is broad, but the species records are sparse and concentrated in only a few areas. This scenario highlights the need for a substantial number of diverse data points to effectively model species distributions, especially in extensive study areas.

Generally, our efforts have demonstrated how model performance responds to various spatial background restrictions and distribution methods (random and biased) in predicting habitat suitability for Rattus species, modelers have the discretion to choose methods and flexibly define the extent of background selection (Merow et al. 2013; Whitford et al. 2024). This approach carries the responsibility of ensuring that the predictive outcomes are theoretically sound, ecologically meaningful, and statistically rigorous. Fourcade et al. (2014) noted that there are no universal guidelines for mitigating sampling bias and enhancing model performance, often due to datasets lacking clear indications of inherent biases. Thus, we suggest that investigating different methods and analyzing their impact on model performance is a good practice. This approach reveals various outcomes under different scenarios and is crucial for making informed decisions about which model predictions are most reliable.