Algorithmic Accountability

Advancements in technology have led to the widespread adoption of machine learning (ML) algorithms in almost all areas of society (e.g., shaping customer experiences through recommendations, and supporting organizational activities through automating tasks). Despite their potential benefits (e.g., personalizing experiences, enhancing productivity, improving decision-making) various examples have shown that systems based on such algorithms can lead to negative consequences. For instance, use cases in healthcare (Obermeyer et al. 2019) or finance (Blattner and Nelson 2021), have demonstrated how ML systems can undermine fairness and discriminate against minorities by generating statistical or reproducing societal biases (Mitchell et al. 2021). Other examples in marketing have illustrated how individuals’ privacy can be compromised by inferring intimidating knowledge about individuals and using it for targeted advertising (Hill 2012; Mattu and Hill 2022). With the development from simple rule-based to complex probabilistic algorithms, the potential for harmful effects gets even more pressing, as their operation is increasingly opaque and more and more automated. While it has been already widely investigated how such issues can be addressed (e.g., Liu et al. 2022; Mehrabi et al. 2021), many organizations still fail to mitigate the often unintended, negative outcomes of the ML systems they are developing, providing, and using. Thus, academics (e.g., Novelli et al. 2023; Wieringa 2020) and policymakers (e.g., Mökander et al. 2022; Smuha 2021) have put an increasing emphasis on the topic of algorithmic accountability to ensure the ethical development and use of such systems (Donia 2022).

Although algorithmic accountability is mentioned as an important principle in almost all guidelines and regulatory documents for the ethical development of ML systems, there is still considerable uncertainty on what constitutes it and how it can be accomplished (Jobin et al. 2019). This is due to the fact, that algorithmic accountability is an ambiguous concept that deals with many different questions (Bovens 2010; Poechhacker and Kacianka 2021; Wieringa 2020). Many of these questions arise from the existence of different accountability types. To shed light on these accountability types and to bring together the different questions around algorithmic accountability, our article aims to introduce the topic to the BISE community. Future work on algorithmic accountability promises to build the foundation for organizations and developers to effectively implement and manage accountability measures and to inform policy-makers on how to appropriately regulate ML systems. Furthermore, it could provide insights into how individuals deal with accountability-related concerns and how it affects their interaction with ML systems.

Algorithmic accountability focuses on the question of who takes the obligation to justify the design, use, and outcome of machine learning systems and who assumes responsibility for the negative consequences of these systems (Bovens 2007; Wieringa 2020). By assigning responsibility and demanding justifications one could oversee and restrain the behaviors of others. Hence, demanding algorithmic accountability can be understood as a governance function to either proactively avoid the negative impacts of the provision and use of ML systems or to reactively sanction accountable actors, if there have been any adverse effects caused by the systems (Novelli et al. 2023). While previous research has extensively studied the requirements that are necessary to achieve the ethical development and use of ML systems (e.g., fairness (Feuerriegel et al. 2020), interpretability (Lipton 2018), privacy (Liu et al. 2022), robustness (Tocchetti et al. 2022)), algorithmic accountability is focused on the demands that shape these requirements and the governance measures that responsible actors can take to fulfill them.

Answering the question of who has to take responsibility for the actions and outcomes of ML systems, the term algorithmic accountability has the potential to mislead. ML systems as technical artifacts cannot be held accountable (Bryson et al. 2017; Martin 2022). However, organizations and users that develop or use these systems could (Martin 2022). As Martin (2019) argued, ML systems are not value-neutral, instead, they integrate the norms and intentions of those developing, operating, and using them. While organizations can design ML systems in a way that gives full control to the users to put the responsibility on them, typically, firms remain in control of the systems they provide. Thus, the normative obligation to be accountable for these systems resides with them and their employees developing and operating ML systems. Concerning the motivation of organizations to act accountable, the accountability concept implies that stakeholders (e.g., regulators) who demand accountability, can put consequences (e.g., economic damage) on accountable actors if they fail to provide justifications and do not assume responsibility (Bovens 2007; Wieringa 2020). Therefore, if stakeholders can successfully enforce accountability demands (e.g., through regulations), firms should have an incentive to avoid the negative effects of their systems and adequately justify their design and use.

Focusing on the question of who are the stakeholders that demand accountability there are typically two types of external stakeholders (Bovens 2007; Wieringa 2020). First, there are users and other individuals that are affected by ML systems (e.g., applicants). These can execute a form of social accountability (e.g., public protest), individually or collectively, to express their concerns and demand justifications and consequences. Second, there are regulators. These are administrative or legal institutions that can implement institutional control mechanisms (e.g., legislation) to require explanations and enforce sanctions.

Following these accountability demands and mechanisms, typically requirements are set that firms have to fulfill to avoid negative consequences. Organizational and technical measures can be used to accomplish this. From an organizational perspective, companies can implement different governance instruments (e.g., establishing and enforcing development guidelines) to hold their employees accountable who are dealing with ML systems (Donia 2022; Schneider et al. 2022). As such, algorithmic accountability can be part of a corporate digital responsibility strategy that is focused on taking responsibility for the ML systems that are developed or used in an organization (Lobschat et al. 2021; Mueller 2022). In addition, developers need to consider certain technical requirements and properties of ML systems that need to be technically managed and addressed to work toward algorithmic accountability by design (i.e., the fulfillment of all requirements that are set by different accountability demands).

Figure 1 illustrates the different stakeholders and the external accountability demands (institutional and social accountability) and internal accountability measures (organizational and technical accountability) that exist around a ML system.

In Fig. 1, we distinguish between provider and operator organizations, since the firm that develops and deploys an ML system is not necessarily the same organization that runs and employs it. Similarly, there are typically different types of practitioners within an organization (e.g., developers, product owners, quality managers, specialists who use ML systems) who deal with ML systems and are in control of their design and use. Whenever this is the case, companies or practitioners among each other hold distributed accountabilities to be responsible for the consequences of the system they develop, operate, and employ. In the following sections, we discuss the different accountability demands and measures in detail.

Despite its increasing relevance, the topic of algorithmic accountability has not received much attention in the BISE community so far. The different accountability demands and measures motivate several empirical and design-oriented research opportunities. Table 1 gives an overview of potential areas for future research organized around the different accountability types introduced in the previous sections.

As explained in Sect. 2.1, social accountability is about users and individuals affected by ML systems holding the providers or operators of the system they use accountable. Thus, research in this area should focus on understanding how different perceptions of accountability arise and how they can influence the interaction and usage of ML systems. Thereby, different perceptions of accountability can be distinguished. Individuals can either perceive that the provider or operator organization is willing or unwilling to be accountable for the system they offer, or they can perceive that they are accountable for certain outcomes or actions when interacting with the system. In both cases, scholars can investigate what are the individual, organizational, and environmental characteristics that shape such perceptions. Furthermore, it can be examined how these perceptions influence diverse dependent variables such as algorithmic acceptance, intention to delegate tasks to an ML system, or organizational trust in the provider and operator. Additionally, further work is required that focuses on the strategies that consumers and professionals use to demand accountability. An important issue is thereby the effectiveness of different strategies and the challenges that consumers and professionals face when demanding accountability. Results in this domain can help to better understand how accountability claims are raised and determined by distinct individual and environmental factors. Moreover, it sheds light on the effects of algorithmic accountability on human behaviors.

Current legislation on ML systems is relatively weak (Mittelstadt 2019). BISE researchers can support regulators in specifying algorithmic policies in several ways. First, scholars can theorize about the long-term consequences of ML systems to advise legal and administrative authorities about upcoming implications. Additionally, it can be systematically examined, which values individuals hold that should guide algorithmic accountability policies (Mason 2021). Furthermore, empirical work can be conducted to analyze the effectiveness of future and existing regulations. Focusing on the impact of algorithmic regulations on organizations, future work can investigate how firms adopt certain legal and administrative requirements and how this affects the development and provision of ML systems. How practitioners act on regulations and inspections and if they see it as a threat or opportunity, are additional questions that can be studied. While policy-related research may not be the core of the BISE community, analyzing such issues from a sociotechnical lens can help to inform regulators and practitioners dealing with algorithmic accountability.

Managing algorithmic accountability within an organization presents the challenge of translating abstract ethical principles into actionable requirements and practical governance measures (Mäntymäki et al. 2022; Mittelstadt 2019). Scholars can support practitioners to tackle this challenge by examining how to successfully establish an ethical-aware culture within a company. Moreover, it can be studied which governance instruments enhance the perceptions of practitioners to be accountable for their work and how this translates to technical decisions and business activities. Taking away the skepticism of employees and enabling them with the necessary resources and capabilities to develop ethically-aligned ML systems, many knowledge- and awareness-related barriers to more accountable organizational practices can be removed (Tomilova 2021). Additionally, researchers can analyze which governance measures are most effective in mitigating risks and how they can create actual business value for corporations. Furthermore, the effectiveness of organizational response strategies to accountability claims can be studied to provide guidelines for practitioners on how they can mitigate the economic and reputational damage that often follows accountability demands. Considering the fact, that typically multiple actors are involved in creating and providing ML systems, future work can investigate how distributed accountabilities are organized between different actors and how this influences their perceived accountability. In this vein, it can be also analyzed how different roles (e.g., management vs. developer) are affected by and deal with (distributed) accountability demands. Generating insights on how firms can implement and manage accountability measures is crucial to support practitioners in managing the potential negative consequences of ML systems.

Although there can be many reasons for algorithmic accountability claims, the technical design is often one of the main factors contributing to the adverse outcomes of ML systems. Researchers have already started to theoretically discuss interpretability requirements that are necessary to develop accountable ML systems (e.g., Felzmann et al. 2020; Kroll 2021). However, it has not been defined and evaluated how they can be implemented to allow different stakeholders to assess the business practices of organizations and practitioners. Taking a design-oriented lens, scholars can build and assess explainability mechanisms and interaction interfaces that fulfill the interpretability needs of distinct institutional or social accountability demands. Focusing on the controllability of ML systems, scholars can examine and propose accountability design principles for different levels of autonomous systems. Moreover, additional work on the different sources of negative consequences of ML systems and the development of technical measures to address them is necessary. This can help to tackle the currently existing robustness, fairness, and privacy challenges. Another interesting area for future research can be to analyze the development practices around the implementation of interpretability and controllability methods. Understanding the challenges of the technical implementation can help to develop better tools and methods to work toward accountability by design.

Overall, it can be summarized that algorithmic accountability is much more than the mere question of who takes responsibility for the impacts of ML systems. Instead, it is about how institutions, organizations, and individuals can govern ML systems and how developers and provider of ML systems can fulfill their accountability obligations. While researchers have already started to propose measures and define requirements to achieve interpretable and controllable ML systems, their concrete implementation and the effects on the ecosystem around them have not gained much attention yet. By defining algorithmic accountability as a governance issue and introducing it from different perspectives we provide a foundation for future research on the topic.