Inter-observer variance and agreement of wildlife information extracted from camera trap images

When remote wildlife cameras became commercially available in the early 1990s (Kucera and Barrett 1993) their favourable characteristics as a stand-alone research tool led to a revolution in the monitoring of terrestrial wildlife. Animals could now easily be recorded non-invasively anywhere in the world (Kucera and Barrett 2011; Meek et al. 2015). Cameras have since been widely deployed to answer management and scientific questions alike, applicable to an immense spectrum of inquiry that ranges from simple species detection to animal-environment interactions and ultimately global biodiversity changes (Swann and Perkins 2014; Trolliet et al. 2014; Steenweg et al. 2017).

Camera traps combine several useful characteristics and, hence, have important advantages over most other terrestrial monitoring tools. They are affordable, enabling more extensive and intensive sampling strategies. Their operation does not require specialised skills or training and they cause little disturbance to wildlife. Cameras can be left in remote areas for prolonged time periods operating almost independently. Camera trapping, thus, is a cost-effective way of improving both detection probability of cryptic species and the number of observations obtained, creating permanent records that yield irrefutable evidence of a wide array of species and their behaviours (De Bondi et al. 2010; Welbourne et al. 2015; Caravaggi et al. 2017; Wearn and Glover-Kapfer 2019; Thomas et al. 2020).

It is no surprise then that camera trap studies have increased substantially in many parts of the world (Meek et al. 2015; Agha et al. 2018), often focussing on more than a single species (Burton et al. 2015). Camera traps are now probably used more than any other research tool in terrestrial wildlife studies. Their popularity and functional versatility, however, create a luxury problem. Large quantities of images are generated, and the relevant data contained in these images still needs to be extracted in a reliable fashion. Since cameras can record quick image successions from a single trigger event, devices deployed in areas of high wildlife activity may easily generate up to 40 000 images per day. It is no longer unusual that research programmes operate camera grids with, sometimes, hundreds of devices across entire landscapes (Swanson et al. 2015; McShea et al. 2016). With vast amounts of images being recorded, the time required to review the available footage also increases, thus demanding more resources to extract data. It follows that most research programmes need to rely on multiple observers to process camera trap information efficiently.

Ongava Research Centre (ORC) in northern Namibia faces a similar challenge. Here, dozens of cameras are deployed regularly on a private reserve to study wildlife ecology and distributions (Edwards et al. 2018; Stratford et al. 2019), to assess camera utility and performance (Stratford and Naholo 2017), and to assist with wildlife management questions on the reserve. For example, over the last decade approximately 50 cameras have been set at permanent waterholes every year as part of the reserve’s wildlife population monitoring programme, routinely generating upwards of 800 000 images within only a few weeks. In order to facilitate the process of tagging, storing, and analysing these magnitudes of image data, ORC developed a software system that provides an optimised user interface. Termed GDMS (Geo-Data Management System), the primary interface for processing images was designed to maximise user efficiency by providing methods for rapid image handling, in particular moving between groups of images. It has been particularly helpful at processing so-called “empty triggers”, image sequences triggered by moving vegetation that contain no relevant wildlife, but can contribute considerably to the number of images needing to be managed. Users record keywords for images, typically species, with simple pre-configured on-screen buttons (Fig. 1). In case of uncertainty, the GDMS enables users to ‘flag’ images for independent verification by other observers. All images and their meta-data (camera types, locations, configurations, etc.) are then stored in a database and can be accessed using a range of filters and sort criteria. This way, ORC has processed > 21.7 million images since 2009, resulting in a database now housing more than 7.6 million keyworded wildlife images that support a variety of analyses. As is customary elsewhere, ORC has depended on multiple data extractors to process and classify image contents.

In principle, the mass-extraction of wildlife information from camera trap images is no different than recording wildlife with any other method. The measurement taken needs to be accurate and sources of recording bias must be minimised. The obvious advantage of camera trapping is that a permanent record exists, one that can be assessed without time constraints in the comfort of an office and be reviewed independently by multiple observers. Despite an objective, static wildlife record, the question remains whether different observers indeed extract the same information from images. Previous research suggests not, showing that inter-observer variance can greatly affect the reliability of results obtained when multiple human observers are expected to perform the same task. Studies found considerable inconsistencies in the information obtained, be it during the identification of mammalian species or unique individuals from camera trap footage (Gooliaff and Hodges 2018; Johansson et al. 2020), or the classification of population characteristics (Newbolt and Ditchkoff 2019). It is also clear that different factors such as camera type (Randler and Kalb 2018) and observer experience can influence the results (Burns et al. 2018; Katrak-Adefowora et al. 2020) and that accuracy may differ with species size and phenotypic distinctiveness (Potter et al. 2019). A variety of mistakes can occur, for instance observers missing species entirely or misidentifying them. Even among trained observers, variance can be high (e.g. Meek et al. 2013), especially if similar-looking wildlife occur in the same dataset (Gooliaff and Hodges 2018).

Different statistical tools are available to measure agreement between observers and account for variance in observer accuracy (Watson and Petrie 2010; Koo and Li 2016; Ranganathan et al. 2017). A random sample of 70 mammal camera trap studies published over the last decade revealed that 43% of articles reported multiple observers extracting information from images for data analysis, yet without measuring reliability between them, while most studies did not explicitly acknowledge whether multiple observers were involved in data extraction, or how this might have influenced the results. Only four studies explained how the risk of misidentifications was mitigated (Supplementary Information 1). The potential value of camera trapping information for biodiversity conservation is beyond doubt (Steenweg et al. 2017; Wearn and Glover-Kapfer 2019). Quantifying inter-observer differences is paramount, however, if the resulting data are utilised for population estimates (Després-Einspenner et al. 2017), species mapping or biodiversity assessments that may lead to conservation policy decisions and prioritisation of limited conservation funds, for example.

In order to understand inter-observer variance in greater detail, we conducted an experiment, asking observers with varying degrees of camera trapping and wildlife identification experience to process the same set of multi-species camera trap images. We aimed to determine inter-observer variance and agreement in terms of: (1) the number of relevant images stored, (2) the number of mammal species identified, and (3) the types and frequencies of mistakes made. We further compared differences in how observers counted herbivore groups and individuals while exploring the influence of observer experience, species distinctiveness and camera location on the results. Finally, we assessed causes of inter-observer variance and how removal of outliers may influence agreement and the accuracy of the results obtained.

Our small-scale experiment raises important questions about the reliability of data extracted by multiple human observers from the same set of camera trap imagery. Observers considerably differed in how they processed imagery, how many data they saved, how confidently they assigned species identifications, how many species they detected overall, and how they interpreted processing rules. Observers also differed in their ability to identify mammalian wildlife accurately, in the number and types of mistakes that occurred and, ultimately, in their estimates of common herbivore individuals and herds seen. Most inter-observer comparisons yielded significant differences and only moderate to good agreement. With few exceptions, the extracted data were characterised by high degrees of variance, with the magnitude of estimate differences ranging in the order of several 100%. These differences are particularly worrying when considering that observers extracted data from near optimal recording conditions; 20 of the 22 focal species had medium to large body sizes (Supplementary Information 4), and the animals were positioned in open areas close to cameras, remaining stationary at waterholes for prolonged periods. Also, we imposed no time limits on data processing and observers had access to a range of resources aiding species identification. Our findings, thus, stress the importance of comparing agreement in multi-observer camera trap studies and, where necessary, the need to truncate unreliable responses.

Observer experience appeared to be one key variable influencing the results. Experienced observers assigned species labels with a very high degree of confidence and accuracy and their herbivore estimates had the highest accuracy overall. They processed images faster and mainly overlooked small-bodied mammals, such as the Rock hyrax, if these were located away from the images’ centre and focal point of animal activity. Less experienced observers, on the other hand, processed data more slowly, yet they assigned species identifications with less confidence, regularly missing and confusing similar-looking mammals, resulting in lower accuracy in species detection and correct identification. Their estimates of herbivore numbers and herds also revealed greater variance. Only using experienced observers for image classification may, therefore, improve accuracy and confidence in the results. However, many research projects rely on (untrained) volunteers for these time-intensive tasks in particular and the number of images obtained during large-scale multi-camera surveys makes it difficult to refer these tasks solely to experienced data processors. It is noteworthy, however, that only one observer, a novice data extractor, detected all 22 mammals present in the entire dataset. During post-experiment feedback, novice and semi-experienced observers clearly pointed out that fatigue was a challenge, especially after multiple hours of continuous data extraction (cf. Oliveira-Santos et al. 2010). This may contribute to an explanation of higher results variance and the number of mistakes that occurred, especially misidentifications. While detecting and counting animals from static images may seem easy, experiments show that human concentration decreases significantly within 30 min of performing intensive cognitive tasks, leading to mental fatigue and increased error rates (Slimani et al. 2018). In our study, observers dealt differently with fatigue. While some stopped image processing and resumed tasks at a later stage (sometimes days later), others continued to finish a particular image sequence. Given the deleterious effects of mental fatigue on human attention (Boksem et al. 2005; Faber et al. 2012), maximum time thresholds for data extraction may be needed to ensure greater reliability. Additional studies are needed to examine the impact of fatigue on camera trapping results.

Most identification mistakes occurred between species with least phenotypic distinctiveness, for instance when observers confused Plains and Hartmann’s mountain zebras. This finding is consistent with other camera trap studies showing that accurate mammal classifications are difficult to obtain where a range of similar-looking species occur sympatrically (Gooliaff and Hodges 2018), or individuals of a particular species exhibit little distinctiveness (Güthlin et al. 2014; Newbolt and Ditchkoff 2019). Following our study, observers commented that animal orientation in the image frame (broad side vs. frontal vs. rear views) as well as ‘crowdedness’, i.e. the occurrence of large numbers of animals in the same image frame, also made accurate identification very difficult. Estimates of Black-faced impala, the species that occurred in the largest social groups at various recording stations, consistently had the highest variance. Group size estimates appeared challenging for any herbivore occurring in gregarious herds, confirming that camera trap-based abundance estimates of non-distinct social species need to be treated with great caution as the cameras’ limited field of view and the complexity of animal movements obstruct an accurate assessment (Stratford and Naholo 2017).

Although based on a small sample of only seven recording stations, our results give evidence that camera location influenced data extraction reliability. This is probably associated with the specific contents and image characteristics recorded at each locality, as background habitat, species compositions and levels of animal activity differed between waterholes. Habitat structure, especially in relation to “openness” has been shown to influence detection and identification accuracy and might have contributed to differing results between camera stations in our study (Wearn and Glover-Kapfer 2019; Egna et al. 2020). Observer image processing rates, mammal detection and identification accuracies, as well as the types and frequency of mistakes indeed varied significantly across the different camera stations. In addition to a diverse range of observer-related influences on classification accuracy, further complications may arise from technical factors influencing image quality, such as image complexity, motion blur and light conditions due to flash settings or varying natural light and contrast between day- and night-time (Rovero et al. 2013; Zheng et al. 2016) (Table 4). Finally, observers also interpreted data extraction rules differently, resulting in greater variance. The novice observer whose estimates frequently contributed extreme outliers adjusted definitions of social group and independent detection events throughout the experiment, indicating that any ambiguity in the data extraction tasks could contribute to higher variance. Inter-observer agreement improved considerably when this participant was omitted from the analyses.

While surveyors can choose from a variety of image management software options to optimise data handling procedures (Young et al. 2018; Greenberg et al. 2019), the final classification of image content still mostly depends on human assessment. Clearly, individual observers can have profound influences on the results. Low agreement between data extractors, however, can lead to skewed population demographics (Newbolt and Ditchkoff 2019) and biased survivorship and population estimates (Morrison et al. 2011; Van Horn et al. 2014; Johansson et al. 2020), thus potentially misguiding conservation measures and policy.

A rapidly evolving suite of automated image processing software aims to mitigate the concerns associated with manual data extraction. These algorithms can overcome certain human biases and issues associated with fatigue, thus improving the precision of results by offering predictable, standardized error rates. However, most, if not all, models still require manual preparation of a training dataset, which is undertaken by both experts (Tabak et al. 2019; Whytock et al. 2021) and/or untrained volunteers (Norouzzadeh et al. 2018, 2019; Willi et al. 2019). Human error is, therefore, not wholly eliminated from the process and at least some of the risks stemming from low classification agreement and false classifications remain. Lower accuracies for out-of-sample data and species with less images in the training dataset (Tabak et al. 2019; Willi et al. 2019; Schneider et al. 2020) suggest that unexpected and rare species may either not be detected or classified incorrectly (Whytock et al. 2021).

Despite fast processing speeds being achieved by some algorithms (e.g. 2 000 images / minute, Tabak et al. 2019), efficiency considerations also need to take into account time and resources required for software development and human classification of training datasets, especially if individual models are not transferable between species and sites. Algorithm classification accuracies and image processing rates in multi-species studies still vary greatly (Supplementary Information 8) and many were similar to those obtained by expert observers using the GDMS. Using models to detect empty image frames and classify common species, however, could significantly reduce the workload of human observers and standardize error rates to predictable levels. Optimizing the processing workflow of large-scale studies with hundreds of thousands or millions of images could involve a hybrid solution, in which trained software models detect animals in image frames but only classify the most common ones, while less common and uncertain records are queried for independent assessment by experienced human observers.

Until machine learning methods can provide reliable automated detection and identification tools for a large variety of common and uncommon species in different habitats (see Willi et al. 2019; Miao et al. 2019; Schneider et al. 2020), much of the time-consuming data extraction work will continue to be the responsibility of human observers. In these studies, the issue of inter-observer reliability can be addressed in different ways. For instance, online on-demand training provides resources to inexperienced data extractors, measurably improving the accuracy of image classifications (e.g. Katrak-Adefowora et al. 2020). Other approaches to reduce the impact of misclassifications may include the omission of records causing disagreement (Rode et al. 2021), multi-observer validation of image content (Swanson et al. 2016), or the selective flagging of doubtful records for independent review by more experienced assessors. Here, we demonstrate that simple outlier screening and truncation can improve reliability. Based on our results, we emphasise the importance of assessing classification agreement in multi-observer camera trap studies, especially if a range of similar wildlife species is monitored simultaneously.

The issue of inter-observer reliability is not unique to camera trap studies. It pertains to any field of research inquiry where multiple human observers are expected to assess the attributes of the same ‘object’ and classify their observations. The problems that arise using multiple observers for data extraction can be highly context specific and, in case of camera traps, a variety of factors influence what humans detect and manage to identify. Our results revealed great inconsistencies in some of the most common measurements obtained with camera trapping: the detection and accurate identification of animals if they occurred (occurrence and biodiversity studies), resulting in both false negatives and false positives, as well as high variances in population size parameters of common social species. Simply put, we do not all see or count the same in a given set of camera trap images, even under favourable conditions. While samples sizes prevented us from testing how the different factors that influence content classification accuracy might interact, our experiment reflected realistic research circumstances in terms of observer selection, the images being processed, the tasks assigned, and the software utilised. Observer performance and associated biases deserve greater attention in multi-species studies, and particularly so if these are founded on mass data extraction by different observers (Cruickshank et al. 2019).