Enter the metrics: critical theory and organizational operationalization of AI ethics

In the last few years, there has been a surge in artificial intelligence (AI) developments and accompanying AI ethical guidelines (e.g. Jobin et al. 2019). Although the importance of ethics to data science is widely recognized (e.g. Herschel and Miori 2017; Boddington 2017), questions have been raised whether the developed discourse on AI ethics is apt to address the currently pressing questions in the field of AI. Especially when it comes to implementing or operationalizing the existing ethical frameworks and methods in real-world contexts, significant challenges remain (e.g. Hagendorff 2020). As such, the general principle-based approach or principled approach, which seeks to form a code of ethics to inform organizations and data scientists on what ethical principles and values should be taken into account to assure ethical development and ethical AI systems, has drawn increased criticism. Recognizing the constructive role of the principled approach in focusing the debate of AI ethics, its limitations have become increasingly apparent (see e.g. Mittelstadt 2019) and concerns are expressed that the approach at this point might even hinder ethical developments in the field of AI, as conflicting ideals and vague definitions can be barriers to the implementation of ethics and to meaningful accountability (Crawford et al. 2019).

This article sets out to answer the question of how the practical impact of AI ethics can be advanced by arguing that the organizational design context requires at least equal, if not more, consideration than the design principles and methods when it comes to understanding the practical embedding of ethics in AI development and deployment. More specifically, the article will build on critical theory, and Feenberg in particular, to argue that the organizational dimension, as the background of values and assumptions that shape the design process, plays a pivotal role in the operationalization of ethics in AI development and deployment contexts. For values and ethical principles to have a meaningful impact on the outcome of an AI system, it is held, a focus on the design context, the organizational systems and processes, and the interests that shape value decisions in the development process is indispensable. With its long tradition of reflection on the relation between human values and the structure of technology, critical theorists like Feenberg can make a valuable contribution to understanding and conceptualizing this dimension of background values and assumptions. In particular, it is held that the critical notions of underdetermination and technical code as defined by Feenberg (e.g. 1991; 2002) can be considered key concepts in expanding the scope of ethical inquiry in AI ethics. The article, therefore, aims to explore how the organizational dimension can be articulated through the work of critical theorists like Feenberg and how increased attention to these implicit contextual aspects of the AI development process can foster the meaningful operationalization of ethics in AI.

Section 2 will briefly discuss AI and algorithmic decision-making and will outline the general developments in the discourse of AI ethics along with its perceived shortcomings. Section 3 provides a brief introduction and background on critical theory, presenting some of its key claims regarding rationality and technology, after which the notions of ‘underdetermination’ and ‘technical code’, as they are developed in the work of Andrew Feenberg, will be discussed. Section 4 relates these notions to the central tensions and trade-offs in the operationalization of ethical principles in AI contexts and will indicate how the contextual and organizational dimension of operationalization plays a pivotal role in the impact of ethics on AI design and employment. Section 5 explores how this approach could contribute to the field of AI ethics and how it might be furthered as an additional way of understanding challenges in the field of AI ethics.

AI, as it is understood today, can be traced back to the convergence of three historical trends (Miailhe and Hodes 2017), (i) the availability of large quantities of data (Big Data), (ii) the possibility of processing this data with growing computer power at relatively constant costs (Moore’s law) and (iii) the achievement in computer science for algorithms to automatically sort out complex patterns out of very large data sets (machine learning). Although there is no generally agreed upon definition of AI, existing definitions are often based on some key functional characteristics of the software, focusing on the system’s interaction with its environment and the capacity to, in varying degrees of autonomy and by computational means, derive decision-relevant outputs such as predictions and recommendations from large datasets of relevant past phenomena from the real world. In the broadest sense, this can include, as the European Commission stressed in her regulation proposal for AI (2021), software developed with machine learning approaches, logic- and knowledge-based approaches and statistical approaches such as Bayesian estimation. A definition based on functional characteristics avoids digressing on future AI developments such as general AI and brings into focus presently existing and ‘near-term artificial intelligence’ (Gunn and O’Neil 2019) that is already in use today.

Given these characteristics of AI, both the appeal and perils of these systems can be addressed. The possibility to train models on vast amounts of labeled and unlabeled data can reduce inefficiencies, improve human decision-making, and optimize organizational processes (Eitel-Porter 2021). Not surprisingly the range of AI applications is vast, going from relatively simple analytics tools to high stake decision-making systems informing or sometimes executing decisions in sensitive domains such as loan applications, child welfare, bail and parole decisions, education, police deployment and immigration (e.g. Whittaker et al. 2018). On the other hand, however, given their widespread use, the scale and depth in which these algorithms can impact individual lives seems unprecedented, making the ethical dimension of these applications immediately salient. The importance of ethics is further stipulated by notable incidents such as the US recidivism prediction algorithm that allegedly mislabeled African-American defendants as “high-risk” at nearly twice the rate as it mislabeled white defendants (see Angwin et al. 2016; or Chouldechova 2016 for discussion), hiring algorithms that, based on analyzing previous hiring decisions, penalized applicants from women’s colleges for technology related positions (Dastin 2018) or a healthcare entry selection program that exhibited racial bias against African-American patients (Obermeyer et al. 2019).

In a first attempt to address these ethical risks of AI “seemingly every organization with a connection to technology policy has authored or endorsed a set of principles for AI” (Fjeld et al. 2020, p. 4). In 2019, as Jobin et al. (2019) found, already over 84 of these frameworks for ethical AI had been published, with the majority published after 2016. This principle-based approach to the ethics of AI, focusing on ethical codes of ethical principles and values that should be taken into account in AI development, determined much of the scope and agenda of governmental policies, academic research on AI and Ethics and of regulatory bodies. In general, these frameworks consist of several key themes, exemplified by the HLEG Guidelines for trustworthy AI that lay out seven key requirements: (1) human agency, (2) technical robustness and safety, (3) privacy and data governance, (4) transparency, (5) diversity, non-discrimination and fairness, (6) societal and environmental well-being, and (7) accountability. Although these frameworks and principles have proven constructive in focussing the discussion and in imposing a shared ethical frame, their proliferation posed a new challenge, as Crawford et al. (2019) remark: “there are now so many ethics policy statements that it became necessary to aggregate them into standalone AI ethics surveys, which attempted to summarize and consolidate a representative sample of AI principle statements to identify themes and make normative assertions about the state of AI ethics” (p. 20). On a meta-analysis level the discourse became partly self-referential by highlighting commonalities and convergence between ethical frameworks for AI (e.g. Floridi et al. 2018).

The implementation and operationalization of these principles, or rather the lack of operationalization opportunities, fueled criticism of the principle-based approach as dominant discourse in the field. From a practical perspective, the principle-based approach seemed toothless, devoid of normative ends and it was remarked that many of the guidelines tilt towards determinism, aligning well with business interests (Greene et al. 2019). Furthermore Mittelstadt (2019) argues that we cannot expect the high-level ethical principles for AI development and deployment to enjoy the same success the principled approach has had in medical ethics as some elemental foundations are lacking (e.g. common aims, fiduciary duties and proven methods to translate principles into practice). Powles (2018) points out that, in the discussions on AI and ethics, there are many diversions (e.g. the existential threat of AI for the human race or the hypothetical possibility of general artificial intelligence) that deflect us from more urgent questions of power in AI. Or, as she states it, we seem to overlook what the state of AI and power is today, how it impacts the questions we ask, and how we might think about it in relation to the questions that we should be asking. Also from a more empirical stance, the approach has been criticized: a study done by McNamara et al. (2020) showed near to zero effect of ethical principles on designer decisions compared to a control group that didn’t read the principles before designing. The focus on principles could even interfere with meaningful change in the industry as for example Wagner (2018) notes that “much of the debate about ethics seems increasingly focused on private companies avoiding regulation where ethics gives meaning to self-regulatory initiatives or even devolves to ‘pre-empting and preventing legislation” (p. 3). So despite the degree of coherence and overlap between the existing sets of principles, without an understanding of the meaningful operationalization and implementation of these principles in contexts of practice, it seems improbable that the principle-based approach as it evolved, is apt to guide ethical development when it comes to the pressing questions of operationalizing ethics in near term AI. A similar challenge can be envisaged for more established principled approaches such as the human right principles that, according to some scholars, should have a more central role in AI strategies (e.g. Hidvegi and Leufer 2019; van Veen and Cath 2018). Even though human rights have a more institutionalized basis and as such have more substance, they remain principle based with similarly limited specifications when it comes to implementation in data science practices.

Additionally, although promising, the more design-oriented approaches to meaningfully operationalize ethics in AI currently seem to equally fall short when it comes to implementing ethical constraints in real life AI contexts. Methods, such as value sensitive design (Friedman 1999; Friedman et al. 2002), designing in ethics (Van den Hoven et al. 2017), ‘ethics in/for/by design’ (Dignum 2018) and the guidance ethics as ‘ethical accompaniment of technology’ (Verbeek 2010), so far still have to find their way to structural embedment outside of the academic sphere. These methods, part of the ‘design turn’ in the philosophy of technology, seek to proactively influence the design of technologies to account for important human values during the conception and design process, where the ethicist “brings in perspectives that might otherwise remain under-represented” (Verbeek 2013, p. 80). They have proven valuable in focussing the ethical discussion on contextualized applications but fall short when it comes to moral guidance on value conflicts and seemed insufficient for ethical system development in real world organizational contexts (e.g. Manders-Huits and Zimmer 2009; Bianchin and Heylighen 2018).

There is good reason to relate insights from critical theorists to the current discussion on AI and ethics as many of the technology related concerns stressed by critical theorists, such as the dominating and controlling role of technology, seem to prefigure current discussions on Big Data and AI [e.g. on ‘surveillance capitalism’ as introduced by Zuboff (2019)]. Moreover, as will be outlined below, their conception of the more implicit and socio-economic dimensions of technology as articulated in the work of authors that are generally related to the field of critical theory, can advance an understanding of the organizational dimension of AI ethics that can complement existing narratives in the field.

Conflicting values or conflicting interpretations of values are far from a new problem in the field of applied ethics (e.g. Berlin 1969; Bianchin and Heylighen 2018). Outlining the central tensions in the operationalization of ethical principles in AI development, however, provides an opportunity to relate the general tensions of operationalizing ethics in AI to the discussion of the ‘technical code’ and the role of the organizational and socio-economic context. From a practical perspective, a crude distinction can be made between two general forms of value conflicts or ethical tensions in the implementation of ethical values in AI: inter-principle tension (the challenge of implementing multiple values and principles in one design) and intra-principle tension (the challenge of translating a single normative principle (in)to a specific technological design). Inter-principle tension arises when, in a practical setting, ethical values place conflicting demands that can’t be satisfied simultaneously [as discussed by e.g. Kleinberg et al. (2017)]. Social or financial services for example, may find themselves in a tension between the need to adhere to the data minimization principle and respect privacy, and their duty of care urging them to maximize data use for the best possible risk profiles and service performance. Additionally, intra-principle tension arises not between ethical demands on a design, but within the operationalization of a single value because there are mutually exclusive (technological) interpretations or because the notion is incommensurable with a technological form. This tension mainly applies to the normative principle of ‘fairness’, which has multiple quantitative definitions [discussed by a.o. Hutchinson and Mitchell (2019); Verma and Rubin (2018)] but is also relevant for operationalizing explainability, privacy and accountability measures. Without discussing these trade-offs and tensions in great length, it is explored how these tensions are related to the technical code to provide a first illustration of how research on design context can advance AI ethics towards the level of operationalization.

As it appears, the current principle-based debate on the ethics of AI and value sensitive methods can play an important role in the awareness of ethical risks but seem insufficient when it comes to fostering the implementation of ethical practices in the domains where AI is actually developed and deployed. Exploring how critical theory, and Feenberg in particular, might contribute to a more praxis oriented ethics in the field of AI, the notions of underdetermination and technical code were discussed to articulate aspects of the organizational dimension of operationalizing ethical principles in AI. This dimension, as was outlined, is an elemental aspect of the adequate assessment and understanding of two general types of ethical tension encountered in the operationalization of ethical principles: inter-principle tension (regarding the trade-off between multiple values or principles in a specific design) and intra-principle tension (the complexity of formalizing a single normative principle due its manifold and mutually exclusive forms of operationalization). Assessing these tensions through the prism of the ‘technical code’ allows for a first conceptualization of this organizational dimension in an AI ethics context. More specifically, Feenberg’s notions of ‘underdetermination’ and ‘technical code’ bring the relation between algorithmic design and the contextual background of values and assumptions shaping design decisions into focus. Ethical AI, it has been argued, requires in addition to sound guiding ethical principles and design methods, an improved understanding of the organizational or contextual background that shapes design decisions and the perceived rationality of design options at hand. Building on critical theory, the inter- and intra-principle tensions can be regarded as upfront ethical challenges that underscore the relevance of the context of operationalization and the hegemony of interest at play in the social dimension of design.

There are some important theoretical and practical insights afforded by the notions of underdetermination and technical code as outlined in this article that go beyond the, perhaps more obvious, point that guidelines do not guarantee ethical AI and that attempts to reshape the agendas these technologies serve will need to address the institutional contexts. For one, addressing and interrogating the organizational context and values as part of the AI ethics operationalization introduces a novel aspect of assessment and implementation that has not received the proper attention in the discourse on ethical approaches so far. Moreover, drawing attention to the organizational context in a way that allows for an engagement with the current challenges in AI ethics concerning the formalization and implementation of ethical principles offers opportunities to develop a more in depth understanding of these challenges and the ways in which they could be met. As these challenges ultimately require awareness, deliberation and strategic decision-making on a socio-economic, political and organizational level, approaches directed towards the meaningful involvement of internal stakeholders over and above the departments and teams tasked with AI development seems necessary. Another way in which these insights surmount the mere underscoring of the role of material interests of people and corporations in AI development, is that, through the prism of the technical code, interests are specified and fleshed out by locating them in the organizational setting of technology development and deployment. Rather than dealing with abstract forces or interests, the value conflicts as addressed in relation to the technical code allow for a contextualization and articulation of specific value trade-offs. Moreover, making this dimension explicit and addressing the value conflicts as socio-economically embedded within an organizational context in turn makes it possible to evaluate the moral underpinnings of specific resolutions or decisions in the value conflicts at hand. Most importantly, however, this conception of the role of ethics in AI development and of what guarantees the best outcome for ethical values in this process, shifts the focus from specific techniques or individual developers towards organizational structures, processes, and business models. It opens a space in the AI ethics discussion to address the relation between the organizational goals, culture and values and the processes of technology development and AI design. Without disregarding the progress that has been made on the (open-sourced) tooling available to assess fairness or explainability of models, an organizational approach brings in scope a critical examination of the contextualized value decisions vis-a-vis AI systems as they are developed within organizations. As AI ethics metrics and assessments might provide guidance on signaling divergence from ethical principles or moral shortcomings in the design of an AI application, the notion of technical code precipitates a discussion on the subsequent questions of governance and procedural checks and balances (e.g. what to do next, how to mitigate the ethical risks, how to organize decision making around ethical risks, who should decide in these matters and how can decisions be morally motivated). Rather than discussing specific use cases or AI applications, the notion of technical code extends the AI ethics discussion towards the ways in which ethical safeguards are embedded in the systems and processes of an organization. As particular critical tools one could conceptualize the instigation of new forms of ethical risk assessment for AI to complement existing technology-oriented assessments. Rather than focusing on compliance with guidelines through checklists, these supplementary assessment methods could focus on relational aspects of ethical risks such as the distribution of risks and benefits between the developing organization and the people subjected to the application. More generally speaking these notions, as developed from a critical theory perspective, advance a more holistic approach to the AI systems and the ethical values they embody, broadening the existing discourse on AI ethics.

From a more theoretical perspective the ideas of underdetermination and technical code can expand and advance the discussion in AI ethics on moral responsibility, organizational incentives and what constitutes value in a more general sense. In questions on the ascription of moral responsibility in AI contexts, for example, the organizational dimension could be a valuable contribution. An oft-mentioned challenge in this field is the responsibility gap, which denotes the widening gap between the behavior of autonomous self-learning systems and their developers and programmers, and the subsequent impossibility to locate the responsibility for bad moral behavior from these systems (see Matthias 2004). Here the inclusion of the organizational dimension and the interaction between socio-economic interests, technology design and background values and assumptions could provide a valuable new approach inciting one to go beyond a reductive individual form of responsibility. As the article provides a first exploration of this theme of organizational operationalization from a critical theory perspective, various other threads for future research remain. The focus on organizational operationalization might provide common ground for other applied ethics disciplines such as business ethics to extend their insights to the AI ethics debate. Since AI is applied in many diverse fields and settings it is not surprising that systematic studies on design contexts and values from an organizational perspective have yet to be fully developed. However, making these interests and their influence on design outcomes in relation to the implementation of ethical principles explicit would significantly advance the field of AI ethics towards practical implementation. As such AI ethics could learn from business ethics and other fields of applied ethics that have already developed an approach to studying the effectiveness of ethical codes (e.g. Kaptein and Schwartz 2008). Second, more research on the relevance of critical theory for the current AI and ethics debate could be fruitful to further develop our understanding of the social impact of AI and the cultural background that guides design processes. The notion of organizational operationalization could propel an inquiry into the broader political discussion of normative ends and contextual appropriateness of certain forms of value operationalization. For example, the technological mediation between AI systems as technical artifacts and human or organizational values, where AI might co-shape our social understanding of common ethical principles (see e.g. Verbeek 2007), is an interesting post-phenomenological thread that could contribute to the furthering of the organizational dimension. Additionally, the emphasis on the embeddedness of AI technologies and AI ethics within a broader political, economic and social context could help spark a more political debate on how interests should be balanced and how contextual appropriateness can be judged. Here a bridge can be made to some monumental works on institutionalized social justice from a.o. Rawls G.A. Cohen and Sen. A focus on the institutional dimension of operationalizing AI ethics allows for a convergence with social justice approaches that have a strong institutional focus and consider society’s economic and political institutions and institutional arrangements as objects of justice.

In an idiosyncratic way, the maturation process the field of AI ethics has to follow could parallel the development of critical theory itself as described by Anderson (2011). Just as critical theory, through the course of its three generations, has been evolving from universalistic principles of morality, justice and truth towards issues of particularity, contextuality, and substantive, non-proceduralistic principles, so too could AI ethics progress from high-level principles towards dimensions of application, contextual justification and judgements of appropriateness.

In addition to a reflection on the current state of AI ethics, this article is meant as a linking pin for the critical theory discourse and the discourse on AI ethics. AI ethics, as it stands, has developed some necessary but insufficient ethics mechanisms to adequately respond to the risks of increased development and deployment of AI. It is the aim of this article to, through the prism of critical theory, expand the scope of AI ethics to include the organizational dimension of background assumptions and values influencing design processes. As a first exploration, much remains to be further developed, nuanced and at times corrected in regard to the challenges and arguments outlined above. However, it should be apparent that critical theory has an important contribution to make when it comes to operationalizing ethical principles and expanding the scope of the discussion on AI ethics towards organizational operationalization. Entering the metrics means confronting the social reality in which AI systems are developed and within which ethical principles have to be formalized. Amidst the rapidly growing deployment of AI systems, the ambition of technical democracy, where design is consciously oriented towards politically legitimated human values, seems more important than ever.