Narratives of epistemic agency in citizen science classification projects: ideals of science and roles of citizens

Humans and ML have the potential to work together in new ways in CS, and make data collection, processing, and validation more efficient (Lotfian et al. 2021; Franzen et al. 2021; Ceccaroni et al. 2019). However, the use of ML raises the question of which tasks will be most affected and which will be relatively unaffected (Brynjolfsson et al. 2018). The distribution and content of tasks according to comparative advantage (who does what best at a given time), to maximize the effectiveness of specialization and increase the efficiency of the system (Kelling et al. 2013), can raise concerns. For example, by making citizen scientists’ contributions either too simple or too complex, or by reducing the range of what they can contribute, there is a risk of disengaging them (Leach et al. 2020). The question of a hypothetical ML takeover of citizen science was also raised by the participants at the 3rd European Citizen Science 2020 Conference (https://​www.​ecsa-conference.​eu/​) during a panel discussion intended to initiate a dialogue on how citizen scientists interact and collaborate with algorithms. As mentioned during the conference event, the current rapid progress in ML for image recognition and labelling, in particular the use of deep learning through convolutional neural networks and generative adversarial networks, presents a threat to human engagement in citizen science; if machines can confidently carry out the work required, then there can be no space for authentic engagement in the scientific process (Ponti et al. 2021). Therefore, as ML and other forms of AI become increasingly used in CS, even more fundamental questions arise as to whether we will continue to need citizen scientists and, if so, how their contributions will change because of automation.

Typically, allocating tasks between humans and machines is related to an increased effort to automate parts of human contributions based on what machines and humans are recognized to be better at (e.g., Fitts 1951) to maximize efficiency and speed to achieve a given goal (Tausch and Kluge, 2020). In CS, the use of ML presents opportunities to improve speed, accuracy, and efficiency in analysing massive datasets, monitoring the results, and identifying knowledge gaps (Ceccaroni et al. 2019). We may conjecture that if project organizers have primarily productivity goals, they may replace citizens as much as possible and only make tasks for them as meaningful as needed to keep them engaged. In contrast, if project organizers also have “democratization” goals, they are going to use machines more for the benefit of human engagement and may even involve citizens even where machines could do a more efficient job. The distinction may not matter much now, because AI is still not capable of replacing people completely. Nonetheless, this distinction can become critical once AI becomes more powerful—then project organizers will have to decide whether they want to maximize efficiency by replacing citizens, or maximize engagement by keeping them in the loop while using machines to make things more interesting for citizens. The distribution of epistemic agency would then be taken into account.

In performing tasks, humans and machines do things, and as a result we can call them actors (those who act). More specifically, they are epistemic actors, because they do things to pursue specific epistemic goals (Ahlstrom-Vij 2013). In this paper, both attaining knowledge and providing solutions to specific problems qualify as epistemic goals. Considering that processes of knowing or problem-solving take place in increasingly entangled systems consisting of human and non-human actors, systems in which data from multiple sources gets processed, accepted, rejected, and modified in various ways by these different actors, the notion of epistemic agency needs to be examined to account for such socio-technical processes. Humans and algorithms are not seen here as self-contained epistemic actors in their own right, and a key empirical question is how agency dynamics play out in hybrid settings combining humans and machines. In this paper, we seek to answer the following questions:

Through the analysis of the narratives reported by organizers (e.g., project leaders, researchers) in the documentation of a sample of projects, we sought to gain descriptive insights into how citizens, experts, and ML participate in specific classification tasks and how task distribution affects their epistemic agency. Narratives are cultural artefacts that tell stories, which offer particular points of view or sets of values (Bal 2009). We consider academic papers and other non-fictional material used in this study as forms of “narratives” offering written accounts of the combination of CS and ML in classification projects.

We will now clarify the terms used in these questions, except for epistemic agency, which will be treated separately in the next section. For a proper understanding of the term task, we refer to Hackman’s (1969, p. 113) definition of the term as a job assigned to a person (or group) by an external agent or that can be self-generated. A task includes a set of instructions that specify which operations need to be performed by a person concerning an input, and/or what goal is to be achieved. We used Hackman’s (1969) conceptualization of tasks as a behaviour description, that is, a description of what an agent does to achieve a goal. Thus, the emphasis is placed on the reported behaviour of the task performer. This conceptualization applies to both humans and machines performing tasks. We chose to use the notion of task for two reasons. First, prior work on CS has also focused on it (e.g., Crowston et al. 2019; Franzoni and Sauermann 2014). Second, as Brynjolfsson et al. (2018) pointed out, the impact of ML on different jobs is likely to depend on the suitability of ML for specific tasks within jobs. Therefore, human participants in CS will be affected differently based on how suitable their tasks are for automation. In this paper, we focus on data-related tasks, such as data collection, processing, and analysis. Therefore, we excluded from our analysis other tasks that experts commonly do in CS projects such as, for example, securing funding, developing materials and methods, and writing papers. Regarding the term “expert”, we use it to include only professional scientists and professionals responsible for developing algorithms, setting up and running the projects. For the sake of our analysis, we do not use the term to refer to “expert citizens”, although we recognize that citizens can develop expertise in CS projects (e.g., Epstein 1995; Collins and Evans 2007). Citizens have been shown to develop expertise and perform tasks that extend beyond the ones they were mobilized with by scientists (Kasperowski and Hillman 2018).

The paper is structured as follows. We begin using a perspective in Science and Technology studies to define “epistemic agency” as a construct that allows exploration of what it means for actors to participate in socio-technical endeavours as CS classification projects. Second, we present the methodology used to collect and analyse our sample, followed by the results and discussion. The final section presents conclusions from this study and suggests future research directions.

The contribution of this paper is threefold: (1) to give scholars studying CS and human–machine integration a synthesis of results providing descriptive insights into the distribution of tasks and epistemic agency in CS classification projects; (2) to draw potential broader implications for the role of citizen scientists that are associated with the division of labour between the three actors; and (3) to point to relevant questions for future research.

We begin by providing an overview of the main characteristics of the sampled projects with examples of tasks performed by citizens, experts, and machines, respectively (Table 1).

Table 2 presents a summary of the three tasks most frequently performed by each actor across the projects. A complete description of all the tasks is included in ESM Appendix 4, along with examples of data for each. The table highlights the role of experts in checking model predictions and validating the results to ensure accurate outputs by models; the role of citizens labelling data to develop a training dataset fed into machines to make correct predictions; and the role of machines inferring patterns from new data after training with a labelled dataset.

We now summarize the dataset according to the major categories and codes aggregated by the number of references (portions of coded text) across the 12 projects. In Figs. 1, 2, and 3, we present the distribution of tasks performed by citizens, experts, and ML across the projects.

Finally, the multiple bar chart in Fig. 4 displays the tasks across the three actors to see the distinct roles of machines, citizens, and experts.

The best example of epistemic agency in the data is provided by Human-in-the-Loop (HITL). Developing and improving ML models without human assistance is not possible yet, therefore, HITL (Shih 2018) is the prevailing approach, which requires human interaction when algorithms encounter problems. Typically, HITL is used to combine human and machine knowledge to create a continuous circle where ML algorithms are trained, tested, tuned, and validated. In this loop, with the help of humans, algorithms become better trained and make more accurate predictions. In other words, at their current stage, ML algorithms can learn and improve on their own through trial and error.

When algorithms provide incorrect suggestions. In Observation.org, a free tool for field observers to record and share their plant and animal sightings, citizen scientists upload images of flora and fauna and if the recognition algorithm fails to provide correct identification of the species, then citizens can edit the wrong suggestion on the observation screen. Based on this, the system shows whether citizens have accepted or rejected observation data. Citizens contribute to creating a sort of gold standard database used to train the ML model (Fig. 5).

For example, in Snapshot Serengeti, machines may misclassify animals in the collected pictures. Citizen scientists are then tasked with identifying the animals and training the algorithms based on their observations. First, the algorithm classifies the picture. If the animal is detected with a certain probability, citizens come onto the scene. AI offers a primary classification (animal recognition) to the spotter (the trapper who uploaded records can also pre-classify the image). A citizen scientist validates/invalidates the pre-classification and an image is not considered as validated until there is at least a 75% consensus (which can be adjusted in the specific project) among all the citizens involved. This is the input for the algorithms.

When algorithms do not know yet how to perform classification. In Milky Way, a system leveraging citizen science and ML to detect interstellar bubbles, citizens identify patterns that machines cannot identify in bubble detection and contribute to building a database. Researchers use the citizen identification output to train ML and build a model of an automatic classifier (Fig. 6).

When algorithms pose queries. In MAIA, an ML-assisted image annotation for the analysis of marine environmental images, an algorithm poses queries to citizens in the form of training data images. Citizens review these images and determine whether they contain objects of interest for classification or not. Then, they manually refine each image with a circle to mark the object of interest in the image, by modifying the circle position or size, so it closely fits the position and size of the object (Fig. 7).

We examined projects in various scientific fields that use CS in conjunction with ML, such as neuroscience, sociology, oceanography, environmental science, botany, life sciences, astronomy, microbiology, and medicine. Despite the variety of scientific fields, the narratives about these projects in terms of project goals, human–machine integration, use of ML, and distribution of tasks between citizens, experts, and machines, all reveal a story of optimization, scaling up, increasing accuracy, and speeding up. These stories underpin a model of data-driven science propelled by ML affordances that make it easier to sift through massive datasets to infer patterns, while renewing inductive reasoning, and making research less hypothesis-driven (Mazzocchi 2015; Hochachka et al. 2012). At the same time, there is also a growing interest in using AI in research as a means of enabling new methods, processes, management, and evaluations (Chubb et al. 2021).

The diverse projects included in this study bring together a stereotypical narrative of how ML enables epistemic agency. Why is this narrative repeated in various scientific fields? Data-driven science seems to unite projects rather than separate them when they are reported for stakeholder audiences, embedding programs of action (Latour 1992), and envisioning a desired imagined future state (Glaser et al. 2021, p. 13). In this way, a scientific future is enacted, namely, one in which high-powered computing capacities enable the utilization of largely inductive ML applications that require less theoretical pre-processing of data (Glaser et al. 2021, p. 10). The shared narrative reflects a unity of science that extends beyond the epistemic agency that shapes these hybrid formats of science. As science opens up to outsiders to train the automated systems of high-capacity machines, outsiders are encouraged to participate as inductive actors, taking part in a program and performing an ideal of science that accommodates them in the research process. This does not imply that the observations made, classified, and used to train machines are not used to test hypotheses. Citizens’ access points, however, enact a process of induction, which tends to unify the sciences more than separate them. When scientists open the way for volunteers to train machines to speed up scientific processes, this is in line with inductivism’s ideals (Kasperowski et al. 2019).

Science has been associated with at least two main epistemic uncertainties known as the problem(s) of induction. Popper (1934) suggests that although some “problems” cannot be solved, they can be more or less successfully managed. First, observations are uncertain and are usually addressed and managed by different technologies, standards, and protocols. Second, finite observations cannot yield universal conclusions. Observers cannot observe everything and protocols in themselves suffer from being based on finite observations. Inundated with data, they will not be able to cope. Nowadays, ML can learn to recognize patterns in classified data that were not integrated into their original design, since the hybrid use of CS and ML when coupled with an abundance of data makes it possible to manage epistemic uncertainty (Popenici and Kerr 2017). Protocols are guaranteed to evolve and maintain their functions as data sets become larger and the epistemic agency of machines increases. However, when outlining an algorithm’s biography (Glaser et al. 2021), it is critical to include how much data an algorithm can handle without losing its explanatory power and its ability to make additional classifications beyond the original gold standard. Once this new data are gathered, it will be time to retrain the machine with the help of experts or citizens.

Science that incorporates outsiders such as citizen scientists in the research process is more likely to unify sciences than to divide them. Citizen science would then reflect a scientific ideal resembling some aspects of the discipline of observations through protocols, which is reminiscent of the logical empiricists’ argument for the purification of science (Kolakowski 1972). While the 1930s called for a purification of science and research, 90 years later the key terms are computer science, artificial intelligence, machine learning, open science, inclusion, mediation, transparency, democracy, and responsible research. In the 1930s, the epistemological concern was that science must close ranks to save itself; it now appears that science is encouraged to open ranks under a banner of openness and unified investigation.

The problem of induction. The three examples of citizen-in-the-loop in Sect. 4.2 suggest that ML is used as much as possible at a given point in time/stage of evolution of the project, and then humans step in to fill in the missing pieces. Depending on the current strength of ML, the temporal order in which machines and citizens take turns to work on tasks is different. If ML provides wrong outputs, then citizens help retrain the model, but if ML is somewhat better and makes suggestions, then citizens correct the model. If ML models work well, then humans take care of undetected exceptional cases. If ML is perceived as producing optimized outputs, citizens may be made redundant. This is a just conjecture for which there is no direct empirical evidence, but it is worth considering. If the outputs of ML are trusted and relied upon by experts, they may influence the socio-technical assemblage that generates the data on which the same ML is trained (Faraj et al. 2018; Pachidi et al. 2021; Lebowitz et al. 2021). It is unclear if humans will ever become redundant when applying ML to the CS domain, and if they do, under what circumstances is ML considered optimal? In the present stage of development, the skills of ML make it a scalable complement to citizens and experts, for example by structuring large amounts of unfiltered data into information or estimating the probability of an occurrence of an event based on input data (Ponti and Seredko 2022).

However, rather than considering optimization from a technical perspective, we relate it to the way the “problem of induction” from observations is handled. A long-standing belief holds those inductive approaches—now emphasised by ML which infers general rules from observations—are plagued by inescapable and insoluble problems (Popper 1934). Humans and machines cannot solve these problems, nor can some sophisticated hybrid approach combine the two. There is an important reason for this: no protocol can handle a lot of data without encountering anomalies, and these must be controlled for. Therefore, human epistemic agency will be needed for never-observed phenomena, because protocols and gold standards have a limited epistemic reach in those cases.

In this study, we focused on technologies using ML classification methods that infer patterns from training datasets consisting of labelled input–output pairs and classify new inputs into predefined output classes. These ML methods are rule-based systems that allow ML to represent experts’ know-what knowledge (Lebovitz et al. 2021). This type of knowledge is captured, for example, in the gold-standard labels used to train and validate ML models. Experts are key in the data-intensive projects we examined. Consonant with the relational notion of agency adopted in this work, we use a relational notion of expertise. Rather than being something experts possess, we define expertise as the expert’s ability to mediate between the production of knowledge and its application (Grundmann 2017). In this sense, in our sampled projects, experts define and interpret situations and set priorities for action. As mentioned in Sect. 4.2, experts are the humans mainly involved throughout the research process and in the loop, when models fail or are unreliable. It has been said that under conditions of epistemic uncertainty, official expertise and lay expertise should not be seen as antagonistic but as complementary (Funtowicz and Ravetz 1990). It is these different types of expertise, including that embedded in ML algorithms, which interact with each other to form an assemblage in a "program of action" (Latour 1992, p.152). Therefore, we suggest viewing ML optimization as a constructed assemblage in which citizens, experts, and machines play different roles and exert epistemic agency at different points in time to pursue CS project epistemic goals.

Trust in ML outputs and redundancy of citizens. In this study, we focused on technologies using ML classification methods that infer patterns from training datasets consisting of labelled input–output pairs and classify new inputs into predefined output classes. These ML methods are rule-based systems that allow ML to represent expert know-what knowledge (Lebovitz et al. 2021). This type of knowledge is captured, for example, in the gold standard labels used to train and validate ML models. It is worth considering that if ML tools are trusted and taken-for-granted, and experts rely on seemingly accurate ML outputs over volunteers (including expert volunteers), these outputs may influence the socio-technical assemblage that generates the data on which the tools are trained (Faraj et al. 2018; Pachidi et al. 2021, Lebowitz et al. 2021). Therefore, to consider machines as unproblematic and their output as immutable mobiles (Latour 1990) implies that citizen scientists will play only a minor role in the long run.

One can speculate that ML in CS will allow citizens to focus on higher-level tasks by automating boring tasks. However, as Franzoni et al. (2021) argued, “to the extent that such work is limited in volume or requires additional knowledge and resources that pose barriers for crowd participants, there is a risk that CS becomes less inclusive by focusing primarily on expert volunteers” (p. 17). They acknowledge the risk that CS could become less inclusive if projects rely primarily on experts. While this may not be a concern from a ‘productivity’ perspective, it may limit CS’s potential to advance the non-scientific goals highlighted by the “democratization view” (Franzoni et al. 2021). Relying mainly on expert volunteers could reduce the diversity of current and future citizen scientists by diminishing their range of motivations and disengaging those citizens who want to contribute to science in their spare time and have fun, help science, or spend time outdoors (Geoghegan et al. 2016). Deriving personal meaning and value from participating is important to citizen scientists, who typically volunteer time and effort driven by intrinsic or social motivations and not for financial compensation (Sauermann and Franzoni 2015). However, how CS projects have to be designed to actually cater to diverse needs and expectations seems to be very much in a conjectural stage (Kasperowski and Hagen 2022). Suggestions to avoid disengagement and redundancy include allowing participants to contribute their task of interest even if the task can be fully automated so that their contribution can help improve ML performance; or incorporating new forms of citizen contributions to fill the gaps created by automation (Lotfian et al. 2021).

While these suggestions can be regarded as motivational by some, they can be seen as “over-engaging”, bordering on the unethical. As part of the debate over the ethical problems evoked by citizen science, there is an issue of “over-engagement”, which means being available for free work for science indefinitely (Kasperowski and Hagen 2022). Therefore, we argue that the opposite of becoming redundant can actually happen with the growth of ML in CS. The problem of induction may repeatedly call upon humans, both experts and citizens, in the loop of the process. The fear of AI and ML creating “undemocratic”, hierarchical, or epistemically stratified projects, must of course be closely observed. However, from the perspective of epistemic agency, hierarchy or epistemic stratification could be said to occur constantly in projects on a microlevel, as different actors are endowed with more temporal epistemic power during the course of a project.

The AI industry is showing interest in developing solutions to global problems using AI in combination with citizen scientists. An example is a recent partnership between a team of IBM data scientists and the UN Environment Programme (UNEP) to overcome the challenges associated with citizen data and create a unified, global baseline dataset for measuring plastic pollution in line with UNEP’s Sustainable Development Goal 14 (Clemente 2020). Sloane (2020) contends that ML extends the agenda of the tech industry, which is focused on scale and extraction. The use of ML may exacerbate an “extractive” approach to citizen participation (Sloane 2020), by which data collection and classification remains the primary way for volunteers to contribute to the scientific goals of CS projects. The increasing use of ML in CS classification projects must therefore be related to issues of power dynamics and inequalities in terms of engagement and retention of volunteer participants.

CS wishes to be inclusive in terms of age, gender, ethnicity, geography and social class. However, CS participation is currently skewed demographically and geographically, with biases in age, gender, ethnicity, and socioeconomic status (Pateman et al. 2021). Participants in long-term projects, such as eBird at the Cornell Lab of Ornithology, have been shown to be predominantly highly educated, upper-middle class, middle-aged or older, and white (Purcell et al. 2012). The results concerning gender composition are mixed; however, some projects show a strong bias towards men (Hobbs and White 2012; Crall et al. 2012; Raddick et al. 2009; Wright et al. 2015). Other studies indicate that some projects offer opportunities for disadvantaged groups not otherwise available (Khairunnisa et al. 2021). There is an ethical imperative to involve a diverse group of participants to inform CS projects and provide access to their benefits (Mor Barak 2020).

CS participation is skewed not only in terms of sociodemographics but also in terms of actual contributions. Most contributions are being made by a few (Seymour and Haklay 2017). This lack of diversity in CS reflects different motivations and capacities and raises concerns about the representativeness of data and whether individual, societal, and environmental benefits are evenly distributed (Pateman et al. 2021). Deriving personal meaning and value from participating can be important to citizen scientists who typically volunteer time and effort driven by intrinsic or social motivations and not for financial compensation (Sauermann and Franzoni 2015). Diversity in participation remains a challenge, which the use of ML may exacerbate. The involvement of ML seems to be a case of what we call “designing for”, where citizens are not integrated into the design process from the beginning, but relied on to make the model (ML design) successful ex-post. It has been suggested that the involvement of citizens through CS, particularly during the research design phase, may help reduce bias in data and training annotations for AI, enable public shaping of AI, and foster a lifelong interest in science (Shanley et al. 2021). The long-standing interest among STS scholars of whether new technologies solve problems or rather manage and move them about; displacing, and making problems and some actors and their agency invisible or redundant; seems to reappear as ML and AI are combined with CS (Glaser et al. 2021).

We are left to wonder whether inequality in participation detracts from the promise to make science more democratic, both in terms of including more diverse people in doing science and in making science better aligned with the public interest (Strasser and Hacklay 2018). However, it would be contrary to some current proponents of CS to claim there are “objective” public interests that science can tap into continuously (c.f. Brown 2009). This standpoint seems too often inform voices from both policy and science, when expectations and social imagery of the availability and readiness of citizens to be mobilized into CS projects is produced. Our suggestion would be that the pursuit of such interests must be viewed as acts of performance, thus they are made and cannot be taken for granted.

The non-linearity of the HITL. In our ANT-inspired view, the constructed assemblage including citizens, experts, and ML is a complex effect resulting from mutual interaction and feedback loops, as exemplified in Figs. 5, 6, and 7. These assemblages can be seen as complex triangular systems of citizens-experts-technology in which relationships and loops need to be repeatedly “performed” by all the actors involved, or the assemblages dissolve. ANT is not the only framework that attends to these connections and interactions. Other theoretical endeavors, such as cybernetics (Wiener 1948), have explored complex feedback processes within networking and self-organization of systems. Both ANT and cybernetics aim to conceptualize complexity, they are both sensitive to the hybrid nature of phenomena, and they both emphasize system effects. However, as pointed out by Fenwick and Edwards (2010), a core difference between cybernetics and ANT is the latter’s orientation toward contingent practices and multiplicity. ANT provides a conceptual framework for analysing how the diverse entities of a classification project—including technologies—take a role through specific situated material–semiotic redistributions of expertise and epistemic agency. Uncertainty characterizes contingent practices. Humans are unlikely to act as “controllers or processors” of classifications in a linear way in the loops, as they may not repeat or reproduce the same exact actions under the same input. Even such fixed things as standards and protocols can be uncertain in practice. The loops we exemplified are not expected to be seamless, whereby ML fails to identify a pattern, citizens identify it, and experts feed the correct answer into the training data. These are expectations and possibilities that ML will perform its tasks, but there is no guarantee that citizens will unquestioningly take the assigned epistemic role and dutifully engage in checking errors, or filling gaps. Nor, for that matter, does it mean that ML itself will comply with experts’ wishes. Checking the correctness of classification can happen in multiple ways, depending on the material–semiotic network in which this task is entangled. Tracing the material–semiotic assemblage of checking data classification could reveal the continuity from one version to the other and thus the simultaneous singularity and multiplicity of the assemblage.

This study has two main limitations. First, we relied solely on secondary sources without incorporating other methods (e.g., interviews) that could help reduce bias and compensate for the dearth of documents and their incompleteness. Our study may be limited by the use of the narratives reported in the documents, which represent the authors’ perspectives. Since most research papers tend to report on successful rather than unsuccessful projects, we are likely to have been exposed to mostly successful divisions of labour instead of those that did not work. Being aware of this potential bias, we have been careful not to use documentary evidence as a stand-in for other kinds of evidence that we could not produce using this method.

Second, our study may be limited by the small number and type of projects examined. The sample we used is purposive. Note that the selected projects reflect those that were documented at a particular moment in time, rather than being a truly representative sample of the population.