1 Introduction
In the last few years, there has been a surge in artificial intelligence (AI) developments and accompanying AI ethical guidelines (e.g. Jobin et al.
2019). Although the importance of ethics to data science is widely recognized (e.g. Herschel and Miori
2017; Boddington
2017), questions have been raised whether the developed discourse on AI ethics is apt to address the currently pressing questions in the field of AI. Especially when it comes to implementing or operationalizing the existing ethical frameworks and methods in real-world contexts, significant challenges remain (e.g. Hagendorff
2020). As such, the general principle-based approach or principled approach, which seeks to form a code of ethics to inform organizations and data scientists on what ethical principles and values should be taken into account to assure ethical development and ethical AI systems, has drawn increased criticism. Recognizing the constructive role of the principled approach in focusing the debate of AI ethics, its limitations have become increasingly apparent (see e.g. Mittelstadt
2019) and concerns are expressed that the approach at this point might even hinder ethical developments in the field of AI, as conflicting ideals and vague definitions can be barriers to the implementation of ethics and to meaningful accountability (Crawford et al.
2019).
This article sets out to answer the question of how the practical impact of AI ethics can be advanced by arguing that the organizational design context requires at least equal, if not more, consideration than the design principles and methods when it comes to understanding the practical embedding of ethics in AI development and deployment. More specifically, the article will build on critical theory, and Feenberg in particular, to argue that the organizational dimension, as the background of values and assumptions that shape the design process, plays a pivotal role in the operationalization of ethics in AI development and deployment contexts. For values and ethical principles to have a meaningful impact on the outcome of an AI system, it is held, a focus on the design context, the organizational systems and processes, and the interests that shape value decisions in the development process is indispensable. With its long tradition of reflection on the relation between human values and the structure of technology, critical theorists like Feenberg can make a valuable contribution to understanding and conceptualizing this dimension of background values and assumptions. In particular, it is held that the critical notions of underdetermination and technical code as defined by Feenberg (e.g.
1991;
2002) can be considered key concepts in expanding the scope of ethical inquiry in AI ethics. The article, therefore, aims to explore how the organizational dimension can be articulated through the work of critical theorists like Feenberg and how increased attention to these implicit contextual aspects of the AI development process can foster the meaningful operationalization of ethics in AI.
Section
2 will briefly discuss AI and algorithmic decision-making and will outline the general developments in the discourse of AI ethics along with its perceived shortcomings. Section
3 provides a brief introduction and background on critical theory, presenting some of its key claims regarding rationality and technology, after which the notions of ‘underdetermination’ and ‘technical code’, as they are developed in the work of Andrew Feenberg, will be discussed. Section
4 relates these notions to the central tensions and trade-offs in the operationalization of ethical principles in AI contexts and will indicate how the contextual and organizational dimension of operationalization plays a pivotal role in the impact of ethics on AI design and employment. Section
5 explores how this approach could contribute to the field of AI ethics and how it might be furthered as an additional way of understanding challenges in the field of AI ethics.
2 AI and AI ethics
AI, as it is understood today, can be traced back to the convergence of three historical trends (Miailhe and Hodes
2017), (i) the availability of large quantities of data (Big Data), (ii) the possibility of processing this data with growing computer power at relatively constant costs (Moore’s law) and (iii) the achievement in computer science for algorithms to automatically sort out complex patterns out of very large data sets (machine learning). Although there is no generally agreed upon definition of AI, existing definitions are often based on some key functional characteristics of the software, focusing on the system’s interaction with its environment and the capacity to, in varying degrees of autonomy and by computational means, derive decision-relevant outputs such as predictions and recommendations from large datasets of relevant past phenomena from the real world. In the broadest sense, this can include, as the European Commission stressed in her regulation proposal for AI (
2021), software developed with machine learning approaches, logic- and knowledge-based approaches and statistical approaches such as Bayesian estimation. A definition based on functional characteristics avoids digressing on future AI developments such as general AI and brings into focus presently existing and ‘near-term artificial intelligence’ (Gunn and O’Neil
2019) that is already in use today.
Given these characteristics of AI, both the appeal and perils of these systems can be addressed. The possibility to train models on vast amounts of labeled and unlabeled data can reduce inefficiencies, improve human decision-making, and optimize organizational processes (Eitel-Porter
2021). Not surprisingly the range of AI applications is vast, going from relatively simple analytics tools to high stake decision-making systems informing or sometimes executing decisions in sensitive domains such as loan applications, child welfare, bail and parole decisions, education, police deployment and immigration (e.g. Whittaker et al.
2018). On the other hand, however, given their widespread use, the scale and depth in which these algorithms can impact individual lives seems unprecedented, making the ethical dimension of these applications immediately salient. The importance of ethics is further stipulated by notable incidents such as the US recidivism prediction algorithm that allegedly mislabeled African-American defendants as “high-risk” at nearly twice the rate as it mislabeled white defendants (see Angwin et al.
2016; or Chouldechova 2016 for discussion), hiring algorithms that, based on analyzing previous hiring decisions, penalized applicants from women’s colleges for technology related positions (Dastin
2018) or a healthcare entry selection program that exhibited racial bias against African-American patients (Obermeyer et al.
2019).
In a first attempt to address these ethical risks of AI “seemingly every organization with a connection to technology policy has authored or endorsed a set of principles for AI” (Fjeld et al.
2020, p. 4). In 2019, as Jobin et al. (
2019) found, already over 84 of these frameworks for ethical AI had been published, with the majority published after 2016. This principle-based approach to the ethics of AI, focusing on ethical codes of ethical principles and values that should be taken into account in AI development, determined much of the scope and agenda of governmental policies, academic research on AI and Ethics and of regulatory bodies. In general, these frameworks consist of several key themes, exemplified by the HLEG Guidelines for trustworthy AI that lay out seven key requirements: (1) human agency, (2) technical robustness and safety, (3) privacy and data governance, (4) transparency, (5) diversity, non-discrimination and fairness, (6) societal and environmental well-being, and (7) accountability. Although these frameworks and principles have proven constructive in focussing the discussion and in imposing a shared ethical frame, their proliferation posed a new challenge, as Crawford et al. (
2019) remark: “there are now so many ethics policy statements that it became necessary to aggregate them into standalone AI ethics surveys, which attempted to summarize and consolidate a representative sample of AI principle statements to identify themes and make normative assertions about the state of AI ethics” (p. 20). On a meta-analysis level the discourse became partly self-referential by highlighting commonalities and convergence between ethical frameworks for AI (e.g. Floridi et al.
2018).
The implementation and operationalization of these principles, or rather the lack of operationalization opportunities, fueled criticism of the principle-based approach as dominant discourse in the field. From a practical perspective, the principle-based approach seemed toothless, devoid of normative ends and it was remarked that many of the guidelines tilt towards determinism, aligning well with business interests (Greene et al.
2019). Furthermore Mittelstadt (
2019) argues that we cannot expect the high-level ethical principles for AI development and deployment to enjoy the same success the principled approach has had in medical ethics as some elemental foundations are lacking (e.g. common aims, fiduciary duties and proven methods to translate principles into practice). Powles (
2018) points out that, in the discussions on AI and ethics, there are many diversions (e.g. the existential threat of AI for the human race or the hypothetical possibility of general artificial intelligence) that deflect us from more urgent questions of power in AI. Or, as she states it, we seem to overlook what the state of AI and power is today, how it impacts the questions we ask, and how we might think about it in relation to the questions that we
should be asking. Also from a more empirical stance, the approach has been criticized: a study done by McNamara et al. (
2020) showed
near to zero effect of ethical principles on designer decisions compared to a control group that didn’t read the principles before designing. The focus on principles could even interfere with meaningful change in the industry as for example Wagner (
2018) notes that “much of the debate about ethics seems increasingly focused on private companies avoiding regulation where ethics gives meaning to self-regulatory initiatives or even devolves to ‘pre-empting and preventing legislation” (p. 3). So despite the degree of coherence and overlap between the existing sets of principles, without an understanding of the meaningful operationalization and implementation of these principles in contexts of practice, it seems improbable that the principle-based approach as it evolved, is apt to guide ethical development when it comes to the pressing questions of operationalizing ethics in near term AI. A similar challenge can be envisaged for more established principled approaches such as the human right principles that, according to some scholars, should have a more central role in AI strategies (e.g. Hidvegi and Leufer
2019; van Veen and Cath
2018). Even though human rights have a more institutionalized basis and as such have more substance, they remain principle based with similarly limited specifications when it comes to implementation in data science practices.
Additionally, although promising, the more design-oriented approaches to meaningfully operationalize ethics in AI currently seem to equally fall short when it comes to implementing ethical constraints in real life AI contexts. Methods, such as value sensitive design (Friedman
1999; Friedman et al.
2002), designing in ethics (Van den Hoven et al.
2017), ‘ethics in/for/by design’ (Dignum
2018) and the guidance ethics as ‘ethical accompaniment of technology’ (Verbeek
2010), so far still have to find their way to structural embedment outside of the academic sphere. These methods, part of the ‘design turn’ in the philosophy of technology, seek to proactively influence the design of technologies to account for important human values during the conception and design process, where the ethicist “brings in perspectives that might otherwise remain under-represented” (Verbeek
2013, p. 80). They have proven valuable in focussing the ethical discussion on contextualized applications but fall short when it comes to moral guidance on value conflicts and seemed insufficient for ethical system development in real world organizational contexts (e.g. Manders-Huits and Zimmer
2009; Bianchin and Heylighen
2018).
Based on the above, one could argue that for ethical principles to be meaningfully implemented in real-world contexts, understanding of ethical principles and design methods are in and of themself insufficient for meaningful implementation of ethics in AI. This article, therefore, holds that the implementation of ethics in AI could benefit from a more in depth understanding of the organizational dimension of operationalization as a third element in the scope of AI ethics. The following sections will further develop this idea that to advance ethical principles and values into actual AI design contexts, along with the design and design principles, special emphasis should be given to the organizational context, as the organizational dimension of operationalization. Ethical decisions regarding model development and deployment are ultimately made within contexts of organizations that have to align the ethical principles with vested interests such as the organizational culture, mission and goals. Critical theory, it is argued, provides a valuable prism for the conceptualization of these organizational aspects of operationalization. In particular the notions of ‘underdetermination’ and ‘technical code’, as they have been developed by Feenberg, can help to bring the contextual background of assumptions, values, definitions and roles guiding design decisions to the fore. The following section will introduce Critical theory and Feenbergs notions of ‘underdetermination’ and ‘technical code’ and outlines their merit in articulating this organizational dimension.
4 Critical theory and AI ethics
Conflicting values or conflicting interpretations of values are far from a new problem in the field of applied ethics (e.g. Berlin
1969; Bianchin and Heylighen
2018). Outlining the central tensions in the operationalization of ethical principles in AI development, however, provides an opportunity to relate the general tensions of operationalizing ethics in AI to the discussion of the ‘technical code’ and the role of the organizational and socio-economic context. From a practical perspective, a crude distinction can be made between two general forms of value conflicts or ethical tensions in the implementation of ethical values in AI:
inter-principle tension (the challenge of implementing multiple values and principles in one design) and
intra-principle tension (the challenge of translating a single normative principle (in)to a specific technological design). Inter-principle tension arises when, in a practical setting, ethical values place conflicting demands that can’t be satisfied simultaneously [as discussed by e.g. Kleinberg et al. (
2017)]. Social or financial services for example, may find themselves in a tension between the need to adhere to the data minimization principle and respect privacy, and their duty of care urging them to maximize data use for the best possible risk profiles and service performance. Additionally, intra-principle tension arises not
between ethical demands on a design, but
within the operationalization of a single value because there are mutually exclusive (technological) interpretations or because the notion is incommensurable with a technological form. This tension mainly applies to the normative principle of ‘fairness’, which has multiple quantitative definitions [discussed by a.o. Hutchinson and Mitchell (
2019); Verma and Rubin (
2018)] but is also relevant for operationalizing explainability, privacy and accountability measures. Without discussing these trade-offs and tensions in great length, it is explored how these tensions are related to the technical code to provide a first illustration of how research on design context can advance AI ethics towards the level of operationalization.
4.1 Inter-principle tension
Value pluralism is a challenge that is often discussed in the form of ethical dilemmas and is a common theme in the fields of business ethics (e.g. Lurie and Albin
2007) and technology ethics (e.g. Wright et al.
2014). Attempting to optimize for all relevant values results in what Bianchin and Heylichen (
2017) call the ‘inclusive design paradox’ where positively improving a system to include as many values as possible might negatively influence the overall application leading them to conclude that “taking human differences seriously seems to imply that nothing can be designed that meets the needs of everyone” (p. 4). When it comes to implementing ethical principles in a data science context, this inevitably has consequences for model development and deployment. Multi-objective trade-offs where two or more desiderata in machine learning systems compete with each other are often inevitable in the operationalization of ethics in AI. For example, keeping models explainable might, on a model level, come at the cost of predictive performance (see e.g. Sokol and Flach
2020). Similarly, studying the implications of implementing fairness constraints in relation to a.o. profit maximization, Hardt et al. (
2016) outlined and quantified how imposing non-discrimination conditions in credit socring and lending has significant implications on profitability. The cost of fairness constraints in relation to the value of safety was studied by Corbet-Davies et al. (
2017), who examined an algorithm for pretrial decisions. They found a tension between the unconstrained and optimal functioning of the algorithm and the constrained algorithm that satisfied prevailing notions of algorithmic fairness. By analyzing data from Broward County they concluded that “optimizing for public safety yields stark racial disparities; conversely, satisfying past fairness definitions means releasing more high-risk defendants, adversely affecting public safety” (p. 8). What studies like these demonstrate is that, in implementing ethical principles, value trade-offs emerge that can and, one could argue, should not be resolved on the level of design and development. Rather they call for contextual and organizational resolutions that require an awareness of cultural and organizational values and how these relate to the ethical principles. This emphasizes the role of policymakers, boards and managers in the ethical development process, as they are the ones ultimately tasked with resolving these organizational challenges. For example, from a cost perspective, in addition to the additional monetary and nonmonetary costs that ethical considerations impose on organizations (Kretz
2016), implementing the ethical constraints in algorithms and data science practices might impose new structural costs on the organization. As of yet, for many organizations it is not clear how to address, assess and decide these sort of value questions and who should make these decisions.
Given the pace and scale with which AI models work, implementing constraints and solutions that are sub-optimal from a profit or efficiency maximizing perspective exposes the organization to performance impairment. Although the importance of these ethical constraints is indisputable, given the technical code and the existing hegemony of interests, one can expect that the background of reductive efficiency oriented assumptions in most organizations, will complicate the full implementation of these constraints. Since organizations, public and private, strive towards profits or efficiency and cost reduction, it is apparent that the operationalization of ethical principles might conflict with organizational interests. The operationalization of ethical principles hinges on the commitment from organizations to value these ethical principles over their primary organizational interests. It could even be argued that, given the background of assumptions and values against which most AI applications are developed in organizations, it is unsurprising that ethical principles and value sensitive design approaches have yielded limited results. Although public interests increasingly move organizations towards the incorporation of public values in the organization strategy (e.g. Frankel
2008), optimization of performance in terms of efficiency, growth or volume is still the dominant imperative, with profitability being the most widely accepted benchmark of overall performance (Daft et al.
2010). Operationalizing ethical principles in AI contexts requires not only a revision of these background values and assumptions in organizations but might also entail a change in how performance of organizations is evaluated.
However, not just the tension between principles can be understood through the prism of the technical code. In a similar fashion, the tension that can occur in translating or interpreting a single ethical principle for AI, the ‘intra-principle tension’, can be shaped by the technical code.
4.2 Intra-principle tension
The underdetermination of technologies and the further elaboration on the technical code imply that the final form of a technological design is not necessarily decided by technical functions but rather by social values and the fit with context. The background values and assumptions are relevant when principles conflict but also in the formalization or implementation of a single ethical principle as for example Binns (
2017,
b) has noted. He points to the need for the formalization of fairness asking what it means “for a machine learning model to be ‘fair’ or ‘non-discriminatory’, in terms which can be operationalized?” (p. 1). As Binns points out, in the field of fair machine learning various different formal definitions of fairness have been proposed, all showing statistical limitations and each incommensurable with the others, indicating that satisfying a single fairness definition might make it impossible to satisfy the other forms of fairness. One could for example opt for equal treatment between groups but that could mean that the overall predictive accuracy for each group is impaired (as there might be legitimate grounds for certain forms of discrimination) (see e.g. Dwork et al. 2012; Angwin et al.
2016). As such, Corbett-Davis and Goel (
2018) hold that it is unlikely that a universally applicable strategy will be found and claim that formal mathematical measures of fairness can inadvertently lead discussions astray. Technical and policy discussions on the interpretation of ethical values, they argue, should be grounded in terms of real-world quantities. The operationalization of fairness in credit risk models should be assessed on the basis of the risk assessment’s immediate and equilibrium effect on community development and the sustainability of a loan program. Ultimately this contextual dimension is what characterizes the operationalization of this value. Binns (
2017) emphasizes this point when he states that “it is clear that the field of fair machine learning faces ‘an upfront set of conceptual ethical challenges; which measures of fairness are most appropriate in a given context?’ (Binns
2017,
b, p. 2).
The tension in the interpretation and formalization of a single ethical principle in AI, necessary because most principles remain ambiguously defined in the principle-based approach, brings the discussion again to the level of the technical code and organizational decision-making. In both forms of tension, inter-principle and intra-principle, the impact and extent to which ethical principles are operationalized are influenced not by specific design values but by the background assumptions and values of the design context. Both forms of ethical tension in the operationalization of ethical principles in AI point towards the importance of the fit between the AI applications in general and the context in which they operate.
5 Discussion
As it appears, the current principle-based debate on the ethics of AI and value sensitive methods can play an important role in the awareness of ethical risks but seem insufficient when it comes to fostering the implementation of ethical practices in the domains where AI is actually developed and deployed. Exploring how critical theory, and Feenberg in particular, might contribute to a more praxis oriented ethics in the field of AI, the notions of underdetermination and technical code were discussed to articulate aspects of the organizational dimension of operationalizing ethical principles in AI. This dimension, as was outlined, is an elemental aspect of the adequate assessment and understanding of two general types of ethical tension encountered in the operationalization of ethical principles: inter-principle tension (regarding the trade-off between multiple values or principles in a specific design) and intra-principle tension (the complexity of formalizing a single normative principle due its manifold and mutually exclusive forms of operationalization). Assessing these tensions through the prism of the ‘technical code’ allows for a first conceptualization of this organizational dimension in an AI ethics context. More specifically, Feenberg’s notions of ‘underdetermination’ and ‘technical code’ bring the relation between algorithmic design and the contextual background of values and assumptions shaping design decisions into focus. Ethical AI, it has been argued, requires in addition to sound guiding ethical principles and design methods, an improved understanding of the organizational or contextual background that shapes design decisions and the perceived rationality of design options at hand. Building on critical theory, the inter- and intra-principle tensions can be regarded as upfront ethical challenges that underscore the relevance of the context of operationalization and the hegemony of interest at play in the social dimension of design.
There are some important theoretical and practical insights afforded by the notions of underdetermination and technical code as outlined in this article that go beyond the, perhaps more obvious, point that guidelines do not guarantee ethical AI and that attempts to reshape the agendas these technologies serve will need to address the institutional contexts. For one, addressing and interrogating the organizational context and values as part of the AI ethics operationalization introduces a novel aspect of assessment and implementation that has not received the proper attention in the discourse on ethical approaches so far. Moreover, drawing attention to the organizational context in a way that allows for an engagement with the current challenges in AI ethics concerning the formalization and implementation of ethical principles offers opportunities to develop a more in depth understanding of these challenges and the ways in which they could be met. As these challenges ultimately require awareness, deliberation and strategic decision-making on a socio-economic, political and organizational level, approaches directed towards the meaningful involvement of internal stakeholders over and above the departments and teams tasked with AI development seems necessary. Another way in which these insights surmount the mere underscoring of the role of material interests of people and corporations in AI development, is that, through the prism of the technical code, interests are specified and fleshed out by locating them in the organizational setting of technology development and deployment. Rather than dealing with abstract forces or interests, the value conflicts as addressed in relation to the technical code allow for a contextualization and articulation of specific value trade-offs. Moreover, making this dimension explicit and addressing the value conflicts as socio-economically embedded within an organizational context in turn makes it possible to evaluate the moral underpinnings of specific resolutions or decisions in the value conflicts at hand. Most importantly, however, this conception of the role of ethics in AI development and of what guarantees the best outcome for ethical values in this process, shifts the focus from specific techniques or individual developers towards organizational structures, processes, and business models. It opens a space in the AI ethics discussion to address the relation between the organizational goals, culture and values and the processes of technology development and AI design. Without disregarding the progress that has been made on the (open-sourced) tooling available to assess fairness or explainability of models, an organizational approach brings in scope a critical examination of the contextualized value decisions vis-a-vis AI systems as they are developed within organizations. As AI ethics metrics and assessments might provide guidance on signaling divergence from ethical principles or moral shortcomings in the design of an AI application, the notion of technical code precipitates a discussion on the subsequent questions of governance and procedural checks and balances (e.g. what to do next, how to mitigate the ethical risks, how to organize decision making around ethical risks, who should decide in these matters and how can decisions be morally motivated). Rather than discussing specific use cases or AI applications, the notion of technical code extends the AI ethics discussion towards the ways in which ethical safeguards are embedded in the systems and processes of an organization. As particular critical tools one could conceptualize the instigation of new forms of ethical risk assessment for AI to complement existing technology-oriented assessments. Rather than focusing on compliance with guidelines through checklists, these supplementary assessment methods could focus on relational aspects of ethical risks such as the distribution of risks and benefits between the developing organization and the people subjected to the application. More generally speaking these notions, as developed from a critical theory perspective, advance a more holistic approach to the AI systems and the ethical values they embody, broadening the existing discourse on AI ethics.
From a more theoretical perspective the ideas of underdetermination and technical code can expand and advance the discussion in AI ethics on moral responsibility, organizational incentives and what constitutes value in a more general sense. In questions on the ascription of moral responsibility in AI contexts, for example, the organizational dimension could be a valuable contribution. An oft-mentioned challenge in this field is the responsibility gap, which denotes the widening gap between the behavior of autonomous self-learning systems and their developers and programmers, and the subsequent impossibility to locate the responsibility for bad moral behavior from these systems (see Matthias
2004). Here the inclusion of the organizational dimension and the interaction between socio-economic interests, technology design and background values and assumptions could provide a valuable new approach inciting one to go beyond a reductive individual form of responsibility. As the article provides a first exploration of this theme of organizational operationalization from a critical theory perspective, various other threads for future research remain. The focus on organizational operationalization might provide common ground for other applied ethics disciplines such as business ethics to extend their insights to the AI ethics debate. Since AI is applied in many diverse fields and settings it is not surprising that systematic studies on design contexts and values from an organizational perspective have yet to be fully developed. However, making these interests and their influence on design outcomes in relation to the implementation of ethical principles explicit would significantly advance the field of AI ethics towards practical implementation. As such AI ethics could learn from business ethics and other fields of applied ethics that have already developed an approach to studying the effectiveness of ethical codes (e.g. Kaptein and Schwartz
2008). Second, more research on the relevance of critical theory for the current AI and ethics debate could be fruitful to further develop our understanding of the social impact of AI and the cultural background that guides design processes. The notion of organizational operationalization could propel an inquiry into the broader political discussion of normative ends and contextual appropriateness of certain forms of value operationalization. For example, the technological mediation between AI systems as technical artifacts and human or organizational values, where AI might co-shape our social understanding of common ethical principles (see e.g. Verbeek
2007), is an interesting post-phenomenological thread that could contribute to the furthering of the organizational dimension. Additionally, the emphasis on the embeddedness of AI technologies and AI ethics within a broader political, economic and social context could help spark a more political debate on how interests should be balanced and how contextual appropriateness can be judged. Here a bridge can be made to some monumental works on institutionalized social justice from a.o. Rawls G.A. Cohen and Sen. A focus on the institutional dimension of operationalizing AI ethics allows for a convergence with social justice approaches that have a strong institutional focus and consider society’s economic and political institutions and institutional arrangements as objects of justice.
In an idiosyncratic way, the maturation process the field of AI ethics has to follow could parallel the development of critical theory itself as described by Anderson (
2011). Just as critical theory, through the course of its three generations, has been evolving from universalistic principles of morality, justice and truth towards issues of particularity, contextuality, and substantive, non-proceduralistic principles, so too could AI ethics progress from high-level principles towards dimensions of application, contextual justification and judgements of appropriateness.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.