The Man Behind the Curtain: Appropriating Fairness in AI

We readily admit that things such as judicial, economic or political systems, institutions, and companies can be qualified as fair or unfair. According to us, we simply need to be aware that we are then predicating fairness analogically, as with the predicate ‘healthy’ applied to animals, drug, urine and food in the example given by Aristotle (1998).¹ Strictly speaking, only the animal can be healthy, and the drug is then called ‘healthy’ insofar as it produces health in the animal, urine is called ‘healthy’ insofar as it is the sign of a healthy animal, and food is called ‘healthy’ insofar as it contributes to the health of the animal. Similarly, a judicial system can perfectly well be called unfair, but only insofar as it reflects unfair actions and beliefs of certain people. The analogical extension of the concept’s meaning can even reach the stage of mere metaphor, for example when we talk about the toss of a fair coin. In this case, the predicate ‘fair’ is also not unrelated to its first usage in reference to people: we speak of a fair coin because the result of the toss is analogous to the result of an impartial person’s random choice between two items. Although admissible, the use of the predicate ‘fair’ is so far removed from its original meaning that it is indeed a metaphor,² and there is no sense in making a normative judgment about the coin, unlike what is possible in the case of an unfair economic system or the responsibility of a juridical person.

On the other hand, many voices in the literature about the philosophy of AI adopt a different approach. Whereas social institutions are conventional in nature and do not enjoy the same sense of individuality as individual human beings, they may be more easily excused from concepts like responsibility or accountability. When assessing the responsibility of corporations for crimes, for instance, moral responsibility is more often than not applied to the managers, CEO, workers, or anyway a set of individual members of the corporation (List & Pettit, 2011). Perhaps because AI devices retain a stronger sense of individuality, researchers are more willing to approach the (un)fairness of AI systems by considering how much agency, autonomy, and responsibility we should attribute to ‘AI agents’.

One important example of this popular approach is the discussion surrounding the ‘responsibility gap’ yielded by AI systems, sometimes called the ‘problem of many hands’ (De Poehl, 2015; De Sio, 2021). Put simply, the issue is the following. Suppose an AI application tasked to automatically assess the creditworthiness of the prospective customer of a financial institution is found to systematically penalize people of color for no real reason. Perhaps, the bias has been hidden in the data set on which the AI application was trained to begin with, and does not represent in any way a real pattern of lower creditworthiness of the people of color requesting access right now. Suppose the behavior of the system is completely unexpected and does not reasonably come from negligence on how it is used. Who is to blame for the unfainess of these decisions? The developers? The company trusting the system? The users? The AI system itself?

There have been different reactions to this problem. To treat them with the necessary attention would be way beyond the scope of this paper. However, a couple of instances will facilitate the point we want to make. One possible approach is to explore the extension of agency and responsibility to AI agents themselves, in order to hold them accountable for their own autonomous decisions (Laukyte, 2017; List, 2021). Another solution would be to pretend that AI systems could be treated as agents, and treat them as if they were the active cause of unfairness under certain circumstances (Tollon, 2023)—for instance, in a similar way to what we do when we treat big corporations as ‘legal persons’, e.g. in the case of limited-liability companies. Again, De Poehl (2015) suggest to deconstruct the problem by capturing the blame for unfair decisions coming from AI systems within the overall ‘socio-technical system’ that made the construction and the deployment of the AI in such a way possible in the first place.

In our opinion, all these solutions have one fundamental limit. They more or less explicitly treat AI systems, due to their relative autonomy in action, as potential loci of agency, responsibility, and (un)fairness. They thus proceed to either include AI agents in their understanding of the world or to see whether and how we can fictitiously ascribe agency to AI or otherwise make sense of why AI seems to be an agent. These all presuppose that, for some reason, we should predicate proper normativity to AI applications or particularly autonomous artifacts, rather than treating them with the same analogical normativity we ascribe to social systems (Table 1).

In the following, we will explore this intuition. We will now introduce some useful conceptual tools to decide whether this popular tendency to attribute proper normativity even in potentiality to AI systems makes sense, or if we should move to an analogical understanding of normativity. After spelling out the conditions for proper normativity in Sects. 3 and 4, we will argue in Sect. 5 that AI systems should be deemed fair or unfair only in analogical sense. The rest of the paper is devoted to pave the way for a treatment of the unfairness of AI along these lines.

Following Arp and Smith (2008), we say that x’s function is a disposition of x that defines its nature, and whose realization is forced by sufficiently favorable conditions in the environment. In other words, an object necessarily tends to realize its function, while it just contingently tends to realize other dispositions. For instance, consider your PC charger. The PC charger is defined by its disposition to charge my computer, so that it may be deemed a good or bad charger based on how well it charges your PC. Thus, charging PC batteries is the PC charger’s function. Now, the PC charger also has the disposition to keep the cat warm as he lays on it. That disposition does not define your PC charger’s nature and is contingent on other features of the environment other than itself. Hence, it is merely a disposition and not a function. A similar notion of function has been linked by Burge (2009) to the emergence of biological normativity.³ This is built into the fact that functions intrinsically manifest how they ought to be performed.

Drawing from Martinelli (2023), we can build a distinction between three levels of agency based on the agent’s relation with its functions.

On a similar note, Ezenkwu and Starkey (2019) distinguish between a high-level form of autonomy and a low-level form of autonomy. High-level autonomy is the ability to set one’s own goals in one’s interaction with the environment, as opposed to merely realizing pre-determined goals imposed externally (Popa, 2021). This is the core difference between tools and fully autonomous agents, if you will (Baird & Maruping, 2021). These notions, respectively, compose the marks of proactive agency and reactive agency. This is because functions operationally represent goals for an agent, as its actions are essentially directed to realize them. To have pre-determined goals amounts to serve built-in functions, like reactive agents do; to be free to self-identify the goals to pursue means retaining the capacity to redirect one’s functions within the environment.

Now, proactive agency, or high-level autonomy if you will, is the condition to interact meaningfully with the environment. By ‘meaningfully’ we follow Di Paolo (2005) and mean that an agent makes sense of the environment if the kind of response the agent gives to a certain input aims at maintaining the internal states of the agent in spite of environmental perturbations. Agents in real-world environments may interact with their environment meaningfully only if they can deal with any kind of perturbation.

As we have already mentioned, normativity itself is already an analogical notion which can be distinguished by different regimes of application. In the next section, we will show that the strict sense of normativity corresponds to the one we use when we evaluate the actions of free and rational individuals, as only these beings genuinely have a relationship with what ‘ought-to-be-the-case’ as opposed to what ‘is-simply-the-case’. However, as suggested by the notion of function above, there is a possible ground for normativity outside the realm of agency alone.

If the heart’s function is to pump blood, we may say that the heart ‘ought’ to pump blood in order to be considered as a heart at all. Even more, a heart may be called good or bad, or, rather, may be said to function well or badly, based on how effectively it pumps blood, i.e. how fully its function is realized (Arp & Smith, 2008). Yet, it is clear that in pumping blood, the heart follows no rules and cannot choose to transgress its function while continuing to be a heart. This kind of functionality devoid of agency is the basis of what we have called ‘analogical normativity’. Artifacts such as houses and chairs ‘must’ be able to fulfill certain functions to be considered as houses and chairs. For example, a house can be red or white, have one or fourteen rooms, one or five floors, but it must (‘ought-to’) in all cases have insulated walls, a watertight roof, and solid foundations to be a house. The difference between a good house and a bad house, e.g. predicated on the quality of insulation and waterproofing (compare a hut in the savannah and a residential villa), is here at best metaphorical and does not have a proper moral character.

The attribution of normative predicates to artifacts is possible because these objects reflect or participate in the actions of normative beings in the primary sense: a good house is a house that reflects the skills of the architect and builders and participates in the unfolding of a good life for those who inhabit it. This comes from the fact that the house fulfills the function that defines it. The analogical extension of normativity from its primary sense to its technical sense concerns this precise point: an artifactual function is analogous to a rule that a rational being follows in performing an intentional action: a good heart is a heart that corresponds to the function that defines it; as in the case of a house, it may be larger or smaller, noisier or quieter, but it ‘ought’ to pump blood to be a heart. The sense of this second analogical extension is as follows: organs and organisms instantiate functions as if they were the product of the activity of a craftsman manufacturing an object to serve a certain purpose.

As technical objects, AI algorithms can thus be integrated into the logical space of technical normativity: they can be good or bad, in an analogical sense of these predicates, depending on whether they fulfill their functions or not. The question is whether they can be qualified as fair or unfair according to a similar analogical extension that goes from the fairness of persons to the fairness of institutions. In the case of the biological entities and artifacts that we considered earlier, they do not seem to be able to be qualified as (un)fair. A heart affected by a pathology could be called a bad heart, but it does not seem possible to say that it would be unfair, even by analogy, to the organism to which it belongs, and similarly for a poorly insulated house that would not be called unfair to its inhabitants. This raises another question, namely what are the characteristics of things that are not single natural persons but can be (un)fair? Before addressing this question, we can already position AI algorithms in relation to the artifacts and biological entities that we mentioned. In this sense, a first thing to note is that biological organs such as hearts and lungs are defined by functions that are not the result of a design, while artifacts such as houses and hammers have functions that are the product of a design. An important difference between these two types of things that instantiate functions, in addition to their relationship to a possible design, is that organs have automatic functioning (they do not require intervention from an external agent to function, they possess active agency in the sense discussed above), whereas an artifact such as a hammer has entirely ‘allomatic’ functioning (it never works without intervention, without direct and constant manipulation by an external agent, i.e. it is entirely devoid of agency). The hypothesis we would like to propose is that we have here the reasons why biological organs and some artifacts cannot be said to be (un)fair although they can be said to be good or bad due to the functions they instantiate. Namely, organs cannot be said to be (un)fair because they do not participate in any design that would be at the origin of their functions, and artifacts like hammers cannot be said to be (un)fair because they are entirely devoid of agency, so it is always the agent who manipulates them who can immediately be held responsible for a (un)fair use of these tools. In both cases, (un)fairness cannot be predicated by analogy because there is no participation: the organ does not participate in any intentional design, and the simple tool does not participate in any action, in the relevant sense of participation, as it immediately is part of the action.

In regards to AI algorithms, we have shown in the previous section that they possess a form of reactive low-level agency. This characteristic sets them apart from simple tools like hammers. However, we have also shown that this form of reactive low-level agency is paired with a form of bounded autonomy that comes from the fact that these AI systems are the product of a design. Therefore, by virtue of this second characteristic, they also differ from biological organs. Do these two characteristics allow us to compare AI algorithms to things that can be said to be (un)fair by analogy? Let’s take the example of an economic system: even in cases of largely planned economies, it can be doubted that such systems are entirely the product of design and it is possible to argue that they always retain a dimension of spontaneous order. It is also not easy to isolate the functions that they instantiate, so it seems difficult to judge their level of agency, or even to say that they possess one regardless of the sense given to this term. However, an economic system can be (un)fair in that it participates in certain (un)fair actions, makes them possible, or manifests them. Unlike tools like hammers, this same system also enjoys a degree of independence from the actions and existence of particular individuals: no individual has to act constantly and intentionally so that the economy works, and the same system can perpetuate itself over many generations, indicating also that it is independent of the existence of particular individuals. Thus, a relationship, or a plurality of mediated relationships to the actions and existence of concrete individuals seems to be a condition for such a system to be qualified as (un)fair. Although AI algorithms may initially appear closer to tools than to economic systems or public institutions, they resemble them precisely in that they also have a degree of independence from the actions and existence of specific individuals and yet always have a mediated relationship to these individual actions and existences.

For this reason, it seems to us appropriate to analogously qualify certain AI algorithms as (un)fair. However, it must be immediately emphasized that this (un)fairness does not imply any form of responsibility on the part of the AI algorithms. Just as it is possible to say that a house is a bad house without attributing responsibility for its flaws to it (while saying that an action is bad is inseparable from attributing responsibility for it to an agent), it is possible to say that an AI algorithm is unfair without attributing any responsibility to it.

The concept of fairness, like all other normative notions, is intimately linked to that of consciousness.⁴ There is no sense in saying that a tool, let’s say a hammer, considered in itself, treats a nail correctly or incorrectly. There are correct and incorrect uses of hammers, but these uses are then attributed to an agent endowed with consciousness and personhood, which ensure that they can be held responsible for their actions. Similarly, one cannot say that a nail has been the victim of injustice because it does not possess any of the relevant attributes for treating a being as a moral patient. Once again, these attributes are mental: setting aside for a moment the question of personhood and agency, at least sensitivity must be attributed to a being to make it a moral patient. In short, and to return to the side of the agent, there is no normativity without mentality: an action, an utterance, or a thought can be evaluated as correct or incorrect, just or unjust, good or bad, fair or unfair only if it can be attributed to a being possessing agency, and most importantly proactive agency as per the previous definition. This being must have reflective consciousness, that is, be capable of forming representations of her own situation in the world. This reflexivity, that is, the ability to have and entertain a sense of self, is also the condition for this being to be held, by herself and by others, responsible for her actions, utterances, and beliefs. The crucial point in what we have just established is that normative agency implies some form of responsibility, which in turn implies that there is a self that can be recognized as the author of its actions, and finally, that this self depends on consciousness. Indeed, if a being were not conscious, in other words, if it were not open to the world, it could not represent its own situation in the world, i.e. it would not be self-conscious. It is in this precise sense that we said earlier that all normative notions are linked to consciousness: it is a condition of possibility for the formation of the self to which responsibility can be attributed.

Two other characteristics of beings that entertain a relation to normativity must still be emphasized: rationality and freedom. A person can only be held responsible for her actions, words, and beliefs on the condition that she can justify them in an intersubjective context and according to certain normative criteria of rationality. For example, an agent would be able to justify her use of a hammer to break a window by saying that it was rational to do so at the moment she did it because she had to face danger and escape as quickly as possible. Conversely, we would not treat someone who systematically failed to provide justifications as a rational and responsible being. In the context of our reflection, rationality is thus grounded in the ability to give reasons when reasons are asked for. Treating a being as a rational and responsible being finally amounts to interpreting what they do and say as not being constrained by purely mechanistic causality. For example, two people could suffer similar bodily harm due to an object falling on their head, but in one case, the object fell due to the wind, and in the second case, due to the deliberate and malicious action of a person holding the object. There is no sense in attributing responsibility to the wind, but obviously, the person who dropped the object is responsible. We not only believe that this person is responsible for the harm suffered by the victim, but also that they acted wrongly because they should not have done that. Finally, if we say that they should not have done that, it implies that they could have done otherwise, meaning that they had genuine alternatives available and they could and should have chosen them. Imputing responsibility therefore implies treating the agent as possessing a significant degree of freedom of action. McDowell (1998) perfectly expresses the ideas presented so far when he imagines what a wolf should possess to be considered a rational being:

A rational wolf would be able to let his mind roam over possibilities of behaviour other than what comes naturally to wolves. ... [This] reflects a deep connection between reason and freedom: we cannot make sense of a creature’s acquiring reason unless it has genuinely alternative possibilities of action, over which its thought can play. ... An ability to conceptualize the world must include the ability to conceptualize the thinker’s own place in the world; and to find the latter ability intelligible, we need to make room not only for conceptual states that aim to represent how the world anyway is, but also for conceptual states that issue in interventions directed towards making the world conform to their content. A possessor of logos cannot be just a knower, but must be an agent too; and we cannot make sense of logos as manifesting itself in agency without seeing it as selecting between options ... This is to represent freedom of action as inextricably connected with a freedom that is essential to conceptual thought.

To sum up: proper normativity appears when beings are endowed with: A corollary is that these beings must be embodied and have a bodily constitution that allows them to be both open to the world and capable of acting in it. For example, a rational being must have a bodily constitution such that, ceteris paribus, when they perceive another person in danger, they draw the correct inference that this person needs help, and act accordingly to help them. A person who, ceteris paribus, perceives the situation but remains immobile or who acts as if to help another person when they perceive no threat would not be considered a rational agent. We therefore have no reason to believe that beings entirely devoid of perceptual openness to the world and motor capacities can possess any form of responsibility. In the next section, we will explore in more detail the specific bodily constitution that an entity must have in order to access normative agency.

Conditions (1) and (2) are the chief ones. In fact, they are the requirements for freedom, rationality, and responsibility to take place. As we take it, for something to act freely, responsibly, and rationally, it must first be able to act and be aware of its actions. (Proactive) agency and (reflexive) consciousness, in their turn, are the two faces of the same coin, as only a sense of self can ground fully-fledged agency in the world, and, vice versa, consciousness is expressed in the performance of the agent’s interaction with its environment (Popper, 1994). However, consciousness and agency are independent insofar as beings can have a mind without possessing conditions (3–5). It should be noted that these latter three characteristics, which are in addition to reflexive consciousness and proactive agency, seem to go together in such a way that one cannot possess one without possessing the other two. As suggested earlier, the beings endowed with all five characteristics must also be social beings, as their rationality only makes sense within social and linguistic practices of justification, of giving and asking for reasons. This conception of normativity reflects ideas developed within a philosophical tradition that ranges from classical American pragmatists and the later Wittgenstein to philosophers of the Pittsburgh School (such as Robert Brandom and John McDowell), passing through Wilfrid Sellars and Donald Davidson. The concepts and theses developed within this tradition are particularly relevant here because they make explicit the criteria we use when we employ normative concepts and when we treat a being as having a rational mind.

As for the question of whether a machine can be conscious, the many positions defended in contemporary debates can be located on a spectrum ranging from the pure and simple impossibility for an artificial being to be conscious to the idea that some machines are already conscious or proto-conscious. According to us, a bad way to approach these questions would be to start from a metaphysical conception of the mind (whether this conception is materialist or dualist) to deduce a set of truths about the nature of physical systems that can or cannot have such a mind.⁵ Therefore, we will not put forward any argument that seeks to a priori include or exclude artificial beings from the realm of beings that can in principle have mental states and, therefore, from the realm of beings that can in principle be held responsible for what they do, say, and think. In the next section, we will establish certain restrictions regarding the bodily constitution that beings must have in order to be able to have mental states, but these restrictions are independent of the artificial nature of the being in question.

Once the questions specific to the metaphysics of mind are set aside, it is possible to approach the question of artificial intelligence and its relationship to fairness from another angle, which we believe is more interesting. That is, from the point of view that begins by asking what criteria we already use to attribute consciousness and responsibility in our daily engagement with the world and the beings that populate it. Or, in other words, what are the presuppositions of the normative point of view that make it possible? Once these criteria are established, as we have done above, it is possible to verify if a being can correspond to them. For example, it makes no sense to apply such criteria in interpreting the movement of a hammer falling to the ground. The hammer does not choose to fall to the ground when it is no longer supported by another object, and all changes in the hammer’s position correspond to this scheme; in short, it is not free and therefore cannot be interpreted as a being responsible for its actions. A conscious being can also be devoid of responsibility: an animal predator killing its prey is not considered to be committing murder, in the relevant sense of the term, because it does not possess the capacities that would allow it to consider alternatives to its actions and to choose between these alternatives based on criteria of rationality. Some movements of rational beings also escape the framework of interpretation in question: the patellar reflex triggered by the impact of a hammer against the kneecap is something that can happen to a rational person, but it is not an action that can be attributed to her. Brandom (2019) provides a useful summary of Davidson’s theory of action by formulating five principles that define the specific type of events that actions constitute: Taking the example used by Davidson (2001), moving one’s finger, turning on the light switch, and thereby alerting a burglar can be described in such a way that they constitute parts of a single event. But while moving the finger and triggering the switch were intentional, alerting the burglar (whose presence was unknown to the agent) was not. What makes an event an action is that there is at least one description under which it is intentional, and what makes a description a description of an intentional event is that it describes the event as the conclusion of practical reasoning. In this example, turning on the light was something that the agent had a reason to do based on a certain goal (e.g. going to the kitchen to get a glass of milk). Although alerting the burglar is something the agent did according to a certain description of the event, it is not part of her action and she could not be held responsible for the burglar’s escape. There is thus a boundary within the world between a type of events to which hammer falls, animal predation, patellar reflexes, and unplanned burglar’s escapes belong, and another type of events that are attributed to agents and evaluated according to norms.

Of course, this distinction between two types of events is not exhaustive. Important subtypes exist within these two categories. Within the events that belong to the first category, i.e., events that are not normative in the strict sense of the term, an important difference exists between events that instantiate functions and those that do not. In this case, as for the principles of action that we have mentioned earlier, one and the same event can be described or specified in many ways according to its causal consequences. For example, it is possible to describe the event corresponding to the beating of a heart in terms of the sounds caused by this beating. However, this description does not allow us to identify the function of the heart, which is to pump blood, not to make noise. As in the case of action, some genuine causal consequences of the event are not relevant to defining the function because they do not belong to its characteristic outputs. A finer classification of events should therefore distinguish between actions, functions, and events that contain neither action nor function. According to such a classification, only actions can be qualified as (un)fair in the primary sense of the term because they are intentional and therefore attributable to a responsible agent. Simple events can only be qualified as (un)fair in an improper or, at best, metaphorical sense, while certain functions can be qualified as (un)fair by analogy under certain conditions.

But for now, let’s stick to the distinction between purely causal events and events involving normativity in the strict sense. What does it mean, more precisely, for an event to belong to the second type? To borrow Sellars’ phrase (Sellars, 1963), when we speak of an action in normative terms, we are not merely describing or explaining it, but we are ‘placing it in the logical space of reasons, of justifying and being able to justify what one says [or thinks, or does]. Any event that cannot be placed in this logical space, the space of reasons, is not an event that can be evaluated in terms of norms, whether they are logical, epistemic, practical, legal, or ethical. And for an event to be situated in this space, it must ultimately be attributable to a person capable of participating in the social and linguistic practices of giving and asking for reasons. We perform these kinds of practices daily: from the courtroom where the judge asks the accused for the reasons behind her actions, to the most trivial of conversations where we ask our interlocutor why they claim their colleague at the office is incompetent. To answer the question of whether an artificial being can be held responsible for a fair or unfair action or decision, all we need to know is whether we can interact with this being and interpret it according to the rules of the game of giving and asking for reasons underlying all our normative practices. Let’s call the claim that only beings who can participate in the practices of interpretation and mutual recognition specific to the space of reasons can be held responsible for their actions the ‘normative pragmatic view’.

There are some kinds of events in any given real-world, unrestrained system that are unpredictable in nature. By ‘unpredictable’ we mean that the content of the perturbation cannot be possibly inferred a priori from the epistemic perspective of the agent. And, in order to respond to new stimuli for which the agent possibly lacks any kind of data to rely on, they need to be able to set their own goals to tackle the situation creatively. In other words, proactive agents may respond meaningfully even to perturbations that evade their present goal’s phase space. For instance, a chess-playing machine would halt if I responded to its last move by just throwing a sandwich on the chessboard. On the contrary, a human player would be able to deal a meaningful response, e.g. asking for explanations, walking away, or proposing new rules to include the sandwich as a legal piece. The difference is that the human player is resilient to a dynamic change happening in the situation at hand (Felin et al., 2013; Landgrebe & Smith, 2022).

The conditions for normativity that we have advanced in the last section imply the capacity to meaningfully orient oneself in the environment—that is, with awareness of the situation, hence rationally, freely, and responsibly. In order for an agent to be normative, it is necessary to understand the totality of the surrounding, including unpredictable events, and to endogenously assign weight to elements of the environment. The process of ‘assigning weight’ to elements of the environment amounts to the capacity of understanding the meaning of what is going on in the surroundings (Boettke & Subrick, 2002). Only by grasping the significance of an event or a new object that enters the agent’s environment for the agent’s own aims, it is possible for the agent to interact with it and to proactively set new goals to further its interests.

The capacity to follow rules is very important for executing predetermined goals in controlled environments. However, complex systems may require the identification of new goals as unpredictable events unfold in order for any agent to function properly, in what is commonly called the process of adaptation (Popper, 1994). Adaptation requires the ability to understand meaning, on top of following instructions (Searle, 1980). Please note that understanding, or the capacity to interact meaningfully with real-world environments like proactive agents do, does not guarantee success in adaptation. For instance, consider a sudden explosion in a shopping mall at rush hour. There is no reason to think any bystander would be able to correctly run for shelter or save other people rather than blundering in the panic of the moment. Anyhow, proactive agents like people would try to come up with solutions to the unpredictable circumstance, by letting go of their previous goals and rearrange their understanding of the environment to identify new goals (e.g. run for that emergency exit as soon as possible). The robot waiter of the local fast food franchise or the robot vacuum of the convenience store would not even try, and would halt on the spot or act at random, simply because that event was not programmed in advance as a possibility in their system.

But what does an entity capable of interacting with complex environments look like? Organisms are the right kind of entity to qualify for proactive agency and thus normativity. We rely on the operational definition of ‘organism’ from (Maturana & Varela, 1980). An organism is an entity composed by functional units (organs) whose functions are concerted to bring about the preservation of the whole. As they put it, organisms are ‘autopoietic machines’ that work for producing parts of the whole and maintining it, rather than producing something that goes beyond the system’s individuality (‘allopoietic machines’). This complex interrelation of functional units is so close that it grounds the organism’s perception of the self as an individual opposed to the external environment.

The fact that a web of functional units is tuned to the composition of a coherent whole allows the organism to overcome, and manipulate, its own functionality. It is this kind of agential structure that endows organisms with the agential dispositions necessary to develop high-level autonomy. This notion of ‘auto-referentiality’ of organisms allows their functions to be endogenously redirected toward the realization of some goal they set in the environment. We see this, in terms of self-identification of goals, in the adaptation of organisms to their environments through biological evolution. Essentially, organisms adapt to their environment in the sense that they can develop new functions by manipulating their faculties and organs. Even at the most rudimentary end of the spectrum, amoebas may elect to wiggle to the opposite side of the primordial soup because they have sensed that a speck of food has dropped.

This action implies something very important: the capacity on the part of the amoeba to set a new goal and pursue it. This has happened in response to an event that was unpredictable a priori. And this self-determination would not have been possible in the absence of an organizational structure oriented to the systematic preservation of the whole. In other words, the array of functions composing the amoeba caused the emergence of a new higher-order function, followed by the organism as a whole, to deal with the new situation (Burge, 2009). While, of course, this sort of agency is too weak to constitute true normativity on the part of the amoeba, it is the organic structure that grounds the preconditions for it.

Put together, these claims are a defense of the so-called ‘organic view’, i.e. the claim that only organisms may qualify for (moral) responsibility (Torrance, 2008). Interestingly for our case, List (2021) has recently defended a similar brand of the organic view by stating that phenomenal consciousness is a necessary condition for moral responsibility, and that (for the moment) only biological life can ground phenomenal consciousness. We find that the organic view is a good starting point for our own claim based on the following reasons.

AI systems are not organisms, and not just because they are not biological entities. They are simply built differently organization-wise, as we shall see in detail in section 5. This has to do with the fact that they are optimizing algorithms, while organisms are brought about by evolutionary processes. As Laukyte (2017) and List (2021) note, this casts a form of ‘bounded autonomy’ onto AI systems—what we would term ‘low-level autonomy’ or ‘reactive agency’. The fundamental reason is that AI agents are doomed to serve predetermined goals, and that in general their attitudes (quasi-beliefs, quasi-desires, quasi-intentions) are determined by the attitudes of other human beings, e.g. the developers that programmed the machine or that designed the system, or the people that generated the data on which the machine is trained. The fact that machines are bound to the phase space inscribed in the dataset they’re trained on or on their program is an indication of this.

Bounded autonomy essentially means that AI systems have a mediated relationship with the pre-intentional world, or what Searle (1983) would call the ‘background of intentionality’. Our intentional states are ultimately grounded on a background of inarticulated, unconscious knowledge that substantiate our direct encounter with the world, composed for instance by tacit know-how or perceptual experiences. Human agents, and organisms in general, have immediate access to the pre-intentional world and directly develop their own attitudes and intentional states. AI systems (and collective agents) only have an access to the pre-intentional world that is mediated by other intentional agents, which dooms them to have attitudes that are explicitly or implicitly funneled by the goals and intentions of others.

The crux of our argument is identifying any deep learning based form of machine learning with optimization. What follows from that identity is that the inherent limits of optimization apply to modern day AIs. Thus, these AIs are significantly limited, especially in terms of freedom and proactivity, which according with our previous section are some of the minimal conditions for normativity. We highlight that we do not claim it is impossible that at some point there will emerge an AI that is not solely based on optimization. However, this is the case at this point in time, at least for any successful AI designs.

Let us start with making the case for interpreting deep learning models as optimization. First, optimization itself is to be understood as a process by which the best object (in terms of a certain objective) is chosen from a set of objects. More formally we follow the definition of optimization provided in (Carissimo & Korecki, 2023) and reproduced here.

While the above definition attempts to be as general as possible, in mathematics and science optimization is often equivalent to a process of maximizing or minimizing a certain function (that is finding its minimum or maximum).

Now, it has been known since the 90s that neural networks are universal function approximators (Hornik et al., 1989). As such, under certain conditions, they are able to approximate a variety of functional relationships between a given class of inputs and outputs. This is the form in which neural networks are employed within AI solutions. Specifically, they are used to approximate some function. This function, depending on the particular flavor of machine learning, might, for example, represent the relationship between states and actions and rewards or the relationship between inputs and intended outputs. By exposing the AI to data or an environment, this function is optimized with respect to a certain chosen value using a variant of stochastic gradient descent (Bottou, 2010). The value chosen might be a certain error between what is intended and what is output or a measure of success often referred to as reward. In concrete terms the deep neural network might, for example, be used to minimize the classification error on a training set in a classification task or maximise the sum of discounted cumulative reward in a reinforcement learning task.

Based on the above we note that the essential component of modern day AIs is a deep neural network approximating some function and the optimization process that is used to maximize or minimize the said function. This process defines and conditions any behavior the AI system might exhibit. It will be behind the values pursued by the AI, the actions it might take and the subsequent inputs that it might expose itself to (assuming it is deployed in an embodied way in the world). But this defining function itself and the objective along which it is supposed to be optimized are not chosen by the AI itself.⁷ Thus, the condition of proactive agency introduced in Sect. 4 is not met as the AI agent, even if it is embodied and able to act in the world, does not have the ability to choose its own goal or modify it. It can always only pursue the pre-determined objective, an immutable property that it has no access to and that in fact defines the crux of its function.

From the lack of autonomy and proactive agency it follows that such an AI agent would also lack freedom. As such, any action it might take must necessarily have been taken by its design and it has no freedom to choose a different action than the one that has been found to be optimal with respect to the pre-defined objective. Of course some AI agents, deployed in the real world, might exhibit certain levels of stochasticity leading to an illusion of choice (as in choosing a different action in conditions which appear the same). This, however, has nothing to do with true freedom as the agent does not have a choice but rather might incorporate a certain pseudo-random component in its optimization process.

It is worth noting that optimization is not a process through which true pro-activity or freedom can be achieved as the objective always needs to be specified a priori. One could imagine introducing meta-objectives based on which the agent then selects a lower level objective giving it an illusion of choice. It is an illusion, because the choice is still defined by the meta-objectives which needs to be specified externally and a priori. An attempt to let the agent specify its own objectives through optimization would lead to an infinite regress of specifying meta-objectives.

Since any deep neural network based AI agent will essentially be an optimization agent it cannot be considered as a normative being due to the lack of autonomy, pro-activity and freedom. However, as outlined in Sect. 2 it is still possible to refer to the AI as unfair by analogy. It is also clear that the responsibility for such an analogously unfair AI lies with different entities than the AI agent itself. In the following section we will attempt to identify the source of that potential ‘unfairness’ in AI.

Each section will present a few edge cases. Their general structure will be to assume the AI is initially ‘fair’. We will define this as follows. We assume fairness can be measured, and is operationalized in a set of fairness-functions J (some examples of such functions as applied to a particular learning setting can be found in Rychener et al. (2022)). These fairness-functions \(j \in J\) can be applied to the data, D, and the objective function of an AI algorithm. An AI is initially ‘fair’ if and only if it is considered fair by every fairness-function \(j \in J\); its creators considered pre-stated criteria of fairness, and ensured that their AI satisfied those criteria. From this assumption, we discuss cases when the AI could still become unfair, thus complicating the attribution of responsibility.

We identify the first source of ‘unfairness’ in machine learning as the data itself or the model choice. Indeed, the choice of the model can lead to significant issues, if e.g. the chosen model is not expressive enough. A good example here would be using a neural network that is not deep enough and so cannot model the intended phenomenon with sufficient accuracy. Moreover, the data used to train a given model has a strong influence on subsequent classifications or actions selected by the model. It has been often stated that the machine learning algorithms are only as good as the data that was used to train them. For one, the data establishes the scope of the algorithm, the space of classes that it can recognize and the inputs that it can be expected to process correctly. If an algorithm trained to distinguish cats from dogs is presented with an image of a human it certainly cannot be expected to function well. Applying a given algorithm to inputs which are out of its scope can lead to ‘unfairness’ stemming from misapplication (this is analogous to using a tool in a wrong way, such as using a hammer to kill a person). The issue nowadays is that due to the incredible amount of data being used in training the scope of the large learning models is not clear even to its designers. Furthermore, even when applied within its scope, the algorithm will still be heavily dependent on the data. If the data itself was biased or unbalanced in any way, this bias will be present in the algorithm itself.

Referring to biases in the data has become very common as of recent. The reason for data being biased can be manifold and we will not give a full treatment of this topic here, but rather point out several examples of such reasons. We refer readers to Ntoutsi et al. (2020) for a full review of the issue of biases in data. As an overview, data can be biased due to the way in which it was gathered (methodological mistake) where for example members of a certain demographic were more likely to provide answers. Here it seems important to note that the concept of bias can be considered as relative and perspective-dependent, where the data is biased with respect to some intended representation. For instance data that can give a good representation of a certain demographic would be expected to mirror statistical properties present in that demographics. Imagine some data represents a female-dominated group. Data points coming from females would be expected to dominate and a data set with a 50–50 split between male and female data points could potentially be biased (in that it would over-represent the male data points). On the other hand if the same data set was to be used in a decision making process it could be considered biased to some extent as the opinion of the minority group (males in this case) would be likely to be under-represented. Here we see that the same data set can be considered biased or not depending on its intended use.

Considering the kind of data many machine learning algorithms are trained on (e.g. huge Internet web-crawls for large language models) they can be considered as a mirror to the particular society that generated the data. Thus, any biases present in the given society will be reflected in the model trained with that data. Similarly, the minority positions and perspectives are likely to be lost if not aligned with the majority’s positions. In this sense machine learning algorithms exhibit the ‘tyranny of the majority’ property present inherently in most democratic systems of governance. When considering the data as the source of ‘unfairness’ it appears clear who is to ‘blame’ for potential lack of ‘fairness’ of the algorithm. While it seems difficult to assign responsibility to the society as a whole the data-induced biases of algorithms are in fact only a reflection of same biases in the data-generating process.

On the other hand, the data used for training is of course selected by a human being or a group of humans (though the process continues to become more and more automatized). As such the people selecting the data also bear some responsibility for making sure it is appropriate and representative for their intended use case. A selective filtering of data could also be a source of biases in the model just as much as the inherent bias of the data-generating process.

The loss or reward function is used to quantify the accuracy of a given classification or action done by the algorithm. In a typical training setting data is presented to the algorithm and it attempts to guess the label of each data point (we will deal with the setting where the agent is embodied and deployed in the Sect. 6.4). The loss is then used to adjust the parameters of the algorithm so that the correct label is guessed or the best action chosen for each training point. The loss function can penalize certain mis-classifications more heavily than others thus potentially introducing a bias (which might be intended or not). Variations of such biased loss functions have been used to address the issue of training data imbalances (Jain et al., 2021). It is quite apparent that biasing a loss function could lead to both fairer outcomes and less fair ones, depending on the specific bias introduced. Thus, a biased loss function could be one source of potential ‘unfairness’.

A more insidious and harder to identify source of ‘unfairness’ comes from a loss function that has not been intentionally biased. For AI models, which are trained using labeled data the loss function depends strongly on the labels themselves. In that case the biases potentially present in the labels would be classified as belonging to the object source as they are part of the data. On the other hand many AI algorithms do not rely solely or at all on labeled data but rather use a reward function to quantify the benefit of certain actions given a certain state. Many such reward functions, selected by the designers, have been shown to lead the AIs into unintended behaviors, which can certainly end up being ‘unfair’. Imagine a traffic signal controlling AI, whose goal is to maximise the sustainability of the traffic system by minimizing emissions. Such an AI actually learns to stop one arm of traffic at an intersection for ever, since the cars generate most emissions when stopping and starting (Korecki et al., 2023). This is an example of reward mis-design or mis-specification. A similar phenomenon, reward mis-generalization, occurs when the AI finds ways of satisfying the reward that were not intended by the designers. AI with such reward are at a very high risk of leading to highly ‘unfair’ situations (as in the traffic signal example, where half of the users of the system will never be allowed through an intersection, while the others get an eternal green signal). In general reward design is a growing area of interest in the AI community and there are many issues associated with it. For one the reward functions are often, in a sense, over-fitting to particular algorithms and architectures making it difficult to realize, if a given reward is applicable generally or only to particular types of algorithms (Booth et al., 2023).

A further issue, that commonly occurs in practice, is that in many cases it is not possible to directly optimize the intended goal function. In such cases a ‘surrogate loss’ is chosen. Examples of such ‘surrogate loss’ could be using grades as a surrogate measure for intelligence or healthcare costs for a proxy for measuring health. The choice of the ‘surrogate loss’ can further lead to consequences, which might be hard to predict at first (Ledford, 2019).

The process source is perhaps the most interesting and least explored source of ‘unfairness’ in AI models and algorithms. As such the process source might involve both the data and the reward but refers exclusively to an agent, which is deployed in an open environment and able to interact with it. If the environment is open and sufficiently complex it becomes impossible to predict a priori what kind of data the agent will be exposed to. If the agent is continuously learning, the data it interacts with might influence its subsequent decisions as well as the subsequent data it is exposed to (feedback loop). In a similar way as the agent explores an open world it might find ways of satisfying its reward function in ways that were not thought of by the designers.

Consider the Microsoft chat-bot Tay (Suárez-Gonzalo et al., 2019), which became racist, sexist and radically politicized within hours of its deployment to Twitter. Tay’s character, unintended by its developers, was the result of a coordinated effort on the part of some Twitter users, who tried to affect it by the kinds of conversations they had with it. As a result Tay has been taken offline after around 16 h of functioning. Moreover, the subsequent chat-bots deployed by different companies have been carefully prepared to be able to withstand these sort of adversarial behaviors on the part of their users [not fully successfully (Borji, 2023)].

The process source points to a highly relevant tension between continuous learning in deployment and the increased risk of developing ‘unfair’ behaviors. Since none of the current AIs understand the concept of fairness (they are only able to optimize for some preset goal) it is impossible for them to judge whether a certain learned behavior is fair or unfair. This further confirms our interpretation of AIs as non-normative eternal optimizers. Therefore, for the AI models that are continuously learning in open environments significant work is performed by the human annotators who are supposed to (among other tasks) eliminate harmful content from the training data (Borji, 2023). Thus the level of openness is somehow restricted by the actions of human annotators (again highlighting that the AI is by no means free).

Here we come back to the statement that we made in Sect. 6 that AI is (un)fair when society is (un)fair. Indeed, an AI deployed in a perfectly fair society, where no one would attempt to skew its models by injecting it with biased data, would likely not need an army of annotators. Alas, our society appears not to be fair and so the AIs that we produce, as much as we might struggle, will end of mirroring some if not all of our own unfairness.