The autonomous choice architect

Choice architecture describes the environment in which choices are presented to decision-makers, or the design of those presentations (Hausman and Welch 2010; Thaler et al. 2012). The logical masters of choice architecture, so-called choice architects, are said to be tasked with the meaningful architecting of choices so as to influence the actions of decision-makers in an intentional way, without restricting options or significantly changing economic incentives (Thaler and Sunstein 2008). The notion of choice architecture emerges as a necessary component of nudge theory, a subset of behavioural science which seeks to meaningfully redesign choice architecture so as to influence decision-makers based on their behavioural biases, without restricting freedom of choice (Sunstein 2014).

Since its popular inception, most nudging and the architecting of choices has been performed by humans, usually in teams arranged as private consultancies or, more frequently, appendages of government (Sanders et al. 2018; Thaler and Sunstein, 2008). Increasingly, however, the architecting of choices is an automated activity, seemingly either devoid of a human choice architect, or devoid of the oversight of a human choice architect (Jameson et al. 2013; Mele et al. 2021; Weinmann et al. 2016; Yeung 2017). Take, for instance, the Facebook News Feed algorithm, which will curate, on a daily basis, around 300 posts to appear to the individual Facebook user, out of an average of around 1500 possible posts (Luckerson 2015). This is nudging via the meaningful design of choice architecture (Johnson et al. 2012); per Thaler and Sunstein (2008), no choices are mandated or banned (a user can always manually go onto pages and see what’s changed), posts are curated because humans are assumed to have bounded cognitive processes (e.g., bias; Simon 1955), and the posts which are selected are those which are predicted to be most interesting to the user (Luckerson 2015).

Insofar as Facebook and similar entities (to name a few: Google and its subsidiary services such as YouTube and Google Maps; Amazon; Microsoft and its subsidiary services such as LinkedIn; Facebook subsidiary services such as Instagram) architect choices, these entities—or rather, the algorithms they design and implement—would seem to function as autonomous choice architects (Johnson 2021; Johnson et al. 2012; Lavi 2017).

In this paper, we will explore the idea of autonomous choice architects, relating the concept to machine learning and AI. The notion that AI is now used to influence individual behaviour in ways both compatible to the concept of nudging and in other ways is not new (Helbing 2015; Jameson et al. 2013; Yeung 2017). However, exactly how AI may play the role of a choice architect is as of yet relatively underexplored (Mele et al. 2021), which is why we go into some detail regarding what nudging is, what AI is, and why we argue that AI can in fact be a choice architect. Having established this, we provide a novel foundation for discussing AI and nudging by avoiding imprecise and arguably erroneous accounts of how these concepts are related.

We define some terms in relation to these ideas in Sect. 2. Equipped with this background, we advance a perspective which regards choice architects as selectors, borrowing from Yeung’s (2017) argument that various technologies, such as machine learning, AI and algorithms more broadly, merely enable “selection optimisation” (p. 121) when it comes to nudging and choice architecture. Indeed, we argue that, at its core, to architect a choice is to select from a set of possible designs that which is predicted to be most effective at influencing the decision of a decision-maker in accordance with a pre-determined objective. This perspective may evoke criticism from various parties. To be clear, our perspective concerns the selection of choice architecture (i.e., nudging) from a set of known choice architectural techniques. Thus, we do not necessarily dismiss the creative and experimental component of designing nudges, which at present remains the reserve of human choice architects; we simply argue that once various components are designed, the process of changing choice architecture (i.e., nudging) is one of selection from this set of designs, a task which can be performed algorithmically or via an AI system. Furthermore, our discussion concerns autonomous systems insofar as they architect choices; we will not generally engage with criticisms of these systems beyond those which arise at this intersection (Susser et al. 2019; Zarsky 2019).

We regard our selection perspective on choice architecture to also be quite intuitive. Consider the popular example of choice architecture where the default option given to a decision-maker is changed. People tend to choose whatever option is given as the default option (Jachimowicz et al. 2019). One striking example of this is given by Johnson and Goldstein (2004), who find that defaulting people into being organ donors, rather than defaulting them into not being donors, leads to around 98–99% of people choosing to become donors, across several European countries. Jachimowicz et al. (2019) offer many further examples in their review of the default option literature, most less dramatic than Johnson and Goldstein (2004) in effectiveness, but still generally demonstrably effective. From a very abstract, theoretical perspective, choice architects who are tasked with setting a default are simply tasked with selecting which option from a set \(\{A, B, C\dots n\}\) should be set as the default (though this is not necessarily a simple task in actuality; Johnson et al. 2012). Traditionally understood, choice architects cannot expand or reduce the number of items in this set, and ultimately will decide to set as the default whatever option is deemed most effective, however this is determined.¹

The perspective of the choice architect as selector does not just apply to the selection of options to be nudged towards, but also to the means of nudging itself (Mele et al. 2021; Mills 2020b). For instance, perhaps—owing to the propensity of bias within a population—changing the default may be found to be a less effective means of architecting choices than, say, using a social norm or a framing prompt. Prior to selecting which option should be nudged towards, therefore, choice architects may also be tasked with selecting which type of design (i.e., nudge) should be used from a set \(\{X, Y, Z\dots n\}\). This stage of selection may not always be possible or required. For instance, where a choice architect is told they can only change the default option, deliberating on alternative types of design is moot (Thaler 2021). Likewise, where technical limitations in terms of the medium of decision making exist (e.g., a paper form, a mobile application), such choices may also be impossible. But this caveat is largely beside the point: whether the question concerns the type of design, or the outcome, or both, choice architects engage most simply is the process of selecting from a set of possibilities.² This perspective broadly follows Mills’ (2020b) framework for personalised nudging.

As with many activities, having reduced the task of architecting choices down to its most basic premise, it becomes reasonable to imagine that this task may be automated, and indeed, as examples such as Facebook’s News Feed algorithm, Google’s search algorithm, and Amazon’s recommendation algorithm all demonstrate, autonomous choice architects already exist (Jameson et al. 2013; Mele et al. 2021). In this paper, we describe the autonomous choice architect as an artificial intelligence (AI). In addition to arguing that AI can take on the role as choice architect, we highlight some of the most obvious implications of this conclusion. A key challenge is that automated choice architecture arguably reduces both transparency and human accountability, and some have even gone so far as to argue that humans cannot bear full responsibility for the actions of complex machines and modern AI (Matthias 2004). While some find cause for a dissolution of responsibility when AI is used, many people not familiar with AI might intuitively hold that it is self-evident that humans remains in charge for whatever tools they choose to use. We agree with such a position, but argue that it is important not to rely on intuition alone when arguing in favour of human responsibility. Both policy-makers and the general public, we argue, benefit from increased knowledge of the nature of autonomous choice architects to effectively argue their case for human responsibility.

The approach we adopt in this article is largely conceptual, as our primary research interest is unpacking the functional parallels between the activities of human choice architects, and the capabilities of autonomous systems, before exploring the consequence of drawing and embracing such parallels. Our objective is to encourage reconsideration of who (or what) the choice architect is, as well as reflection on how the shifting form of the choice architect (from human to machine) changes the responsibilities of the choice architect. However, this article does benefit from existing alongside a growing body of critical research into the intersection of behavioural science and information technology more broadly. For instance, for a more empirical approach to similar questions raised in this article, see Mele et al. (2021). Therefore, particularly in Sect. 3, we will draw on such research and examples to evince and complement our overall conceptual assertion that the choice architect operates as a selector, and may be understood as viably operating autonomously, taking the form of an algorithmic or artificially intelligent system.

In Sect. 2, we offer several definitions which underpin our use of the term AI, and provide a basis for differentiating an autonomous choice architect from a traditional, human choice architect. In short, the position adopted in this paper is that the autonomous choice architect automates the selection and implementation of choice architecture (i.e., automatic nudging or smart nudging; Mele et al. 2021). In Sect. 3, we explore some examples of actual and proposed autonomous choice architects, identifying the similarities these examples share in terms of choice architecting as a selection process, and in terms of operating independent of direct human oversight. In Sect. 4, we consider the implications of autonomous choice architects, arguing that despite the appearance and autonomy of autonomous choice architects, this cannot and should not result in one ignoring the human actors who design, implement and control these systems. We conclude by discussing the implications of autonomous choice architects on practitioners and society.

To begin our discussion, it is useful to provide a brief outline of some of the key terms which will be used throughout. We consider this exercise useful for two reasons. Firstly, terms such as ‘artificial intelligence’ and ‘behaviour’ are often used in a variety of contexts, and as such, can come to capture different concepts depending on one’s perspective (Possati 2020; Turkle 2004 [1984]). For instance, in a recent review of artificial intelligence for the EU Commission, Samoili et al. (2020) identify 55 different definitions of AI. Secondly, a key aspect of our discussion is how humans and AI are similar, and thus how an AI system can come to operate as a choice architect. Thus, it is helpful to proceed with terms that establish equivalency between humans and machines, as well as terms which reveal important differences between humans and machines. This may produce terms some human practitioners take objection to, for instance, the notion that architecting choices is a mere selection task, rather than involving human creativity and imagination. Nevertheless, insofar as various domains now employ AI systems and algorithms to architect choices (Johnson 2021; Lavi 2017; Mele et al. 2021), it is unhelpful in understanding these entities to ignore conceptually useful points of similarity between humans and machines on the basis that they are not perfect descriptions.

To begin, we define behaviour, in terms of the choice architect, as actions in response to stimuli. Furr (2009, p. 372) has offered a more specific definition of behaviour, that being, “verbal utterances or movements that are potentially available to careful observers using normal sensory processes,” yet this definition has been criticised as potentially being out-of-date given the rise of new technologies (Rauthmann 2020). For instance, specification for behaviour to be bounded by “normal sensory processes,” or limited to only “verbal utterances or movements,” is questionable with the emergence of fMRI machines which can be used to observe neural events, or the widespread adoption of smartphones which can track eye movements and subtle facial expressions (Valliappan et al. 2020). Our definition of behaviour is much more in keeping with perspectives from the world of AI (Silver et al. 2021; Russell 2019; Simon 1994 [1969]) which themselves have drawn from earlier theoretical models of human behaviour found in psychology (Hayek 1952; Miller 2003; Turkle 1988; von Neumann 2000 [1958]).

Where Furr’s (2009) definition is more useful for our purposes is the notion of actions being “potentially available” to be observed. Following Furr (2009), just because an action could have been observed, but is not observed, does not mean that the action is not a behaviour. Insofar as the perspective we adopt throughout this paper is one of selectors choosing from sets, the notion of potential actions is desirable as it suggests an observed behaviour is just that which has been ‘selected’ from a set of possible actions by some criteria (Skinner 1976 [1974]). Therefore, we regard behaviour as potentially observable actions in response to stimuli.

We define intelligence, following Russell (2019), as the process of selecting actions which are expected to achieve one’s objectives. Such a perspective is immediately useful in relation to our definition of behaviour. For instance, not all behaviours are necessarily intelligent. Listening to music or waving out of a window are simply actions. But if one’s objective is to rehearse for a music recital, or to escape a burning building, these respective actions may be considered intelligent insofar as they are expected to achieve one’s objectives (Russell 1997). Such intelligent behaviour may also be called rationality of some kind (Russell 1997; Schafer 2018).

The question of how an entity forms their expectations is an interesting one, and points to an important difference between humans and (existing) AI systems. de Vos (2020, p. 3) distinguishes between machines and humans based on humans’ “strange capacity to reflect upon themselves,” which one might describe using another word: learning (also see Ashby 1978; Turkle 1988). Yet the way humans learn is quite different from that of contemporary machine learning systems (Watson 2019), even if the former has inspired the latter (Silver et al. 2021; Simon 1981; von Neumann 2000 [1958]). For instance, humans do not need to see 10,000 images of a cow to correctly identify a cow, whereas AI systems currently do require such intensive training (Watson 2019) in the form of reinforcement learning (Silver et al. 2021). Equally, such intensive machine learning processes can produce results which exceed human results, even when a human is professionally trained in a domain. For instance, McKinney et al. (2020) report results of an AI significantly outperforming medical professionals in identifying cancerous tumours.

This leads us to our final definition, that of a machine. Marx (2013 [1867]) has offered an initial perspective on a machine, arguing it differs from a mere tool by possessing its own “motive power” (p. 257). A tool, by contrast, must be given its motive power by some other entity, usually a human. Such a perspective can also be found in more recent literature (Gunkel 2020; Turkle 2004 [1984]) and is a useful perspective to adopt for two reasons. Firstly, it provides a basis for drawing some equivalency between a human and machine—both possess their own motive power. Secondly, it invites one to ask the question, “why might one replace a human with a machine?” by providing a basis for such replacement (or automation), namely, the ability to substitute motive powers.

To answer to this question, it is because the motive power of a machine is materially different to that of a human (Marx 2013 [1867]). For instance, no human (or any number of humans) could decide which Facebook post to show a user within the time it takes for the app to load in any intelligent fashion (Gunkel 2020; Zuboff 2019). Likewise, it would likely take many weeks and several professionals to collect, analyse and output a judgement on which policy to pursue, or which medical treatment to offer, when thousands of variables need to be considered (Aonghusa and Michie 2021; McKinney et al. 2020). If one describes this material difference as organic (i.e., human) versus inorganic (i.e., machine), we define a machine as an inorganic entity possessive of its own motive power (Table 1).

In this article, we assert that, at its most basic, descriptive level, the role of a choice architect is to act as a selector, firstly in terms of how a person is influenced (i.e., which nudging strategy is used), and second in terms of what a person is influenced to do (i.e., what outcome/option/preference/behaviour etc. a decision-maker s nudged towards; Mills 2020b).

The behaviour of the choice architect (which should not be confused with the behaviour of the decision-maker) is assumed to be orientated towards an objective, and indeed, there is no example of conscious nudging which we know of whereby the choice architecture used was not part of a strategy for achieving an objective. This assumption may stand regardless of whether the nudge conforms to the typical ethical standards of nudging, whereby decision-makers are expected to be left better off by the nudge, while retaining freedom of choice (so-called libertarian paternalism; Thaler and Sunstein 2003). For instance, a vendor may try to nudge a decision-maker to buy a product the decision-maker will not especially enjoy, but that will be profitable for the vendor (Beggs 2016; Lavi 2017). The vendor, therefore, nudges with an objective, though not necessarily for the decision-maker’s benefit.

Accepting this assumption, the behaviour of the choice architect can be described as intelligent, insofar as the act of architecting choices in a particular way is expected to achieve the objectives of the choice architect. Choice architects also learn, or are expected to learn, from previous experiments with changing choice architecture. Some recent reviews include Della Vigna and Linos (2020), who review the effectiveness of nudge randomised-controlled trials (RCTs) across two so-called ‘nudge units;’ Beshears and Kosowsky (2020) who review 174 nudge studies to evaluate the average effect size of different nudge strategies; and Jachimowicz et al. (2019), who review 58 studies specifically investigating the default option nudge to determine the effect size associated with this specific nudge. There have also been recent calls to consider experimental practices in choice-architectural design (John 2021), to consider strategies for scaling nudge interventions (Al-Ubaydli et al. 2021), and for widespread adoption of A/B testing methods (Benartzi 2017).

Human choice architects are also autonomous insofar as they have their own motive power. Yet, as the history of the computer and data collection attests, the material difference in motive power between humans and machines means that humans may be displaced by machines (Zuboff 1988). Indeed, one key consideration for the automation of previously human processes is the sophistication of the task (Brynjolfsson and McAfee 2014). Tasks which can be routinised and described in terms of computational (i.e., mathematical) logic are often prime candidates for automation, and as Simon (1994 [1969]) has argued, the abstraction of a task into a series of instructions often leads to more efficiency (e.g., faster results, more accurate results). The description of the choice architect as a selector is purposely used because it is a description which lends itself to a discussion of automation, while the arguments provided above regarding the efficiency of automated, often algorithmic processes, is also increasingly drawn upon in the discussion of architecting choices (Aonghusa and Michie 2021; Mele et al. 2021; Yeung 2017).

Consider the following example. One common objective which choice architects hold is achieving greater efficiency from nudging, usually understood to mean more people achieving outcomes which leave them, “better off, as judged by themselves” (Thaler and Sunstein 2008, p. 8). Owing to the computational limitations of human choice architects, most nudges must adopt a ‘one-size-fits-all’ approach, which may be effective on aggregate but may not be optimal (Mills 2020b; Sunstein 2012). A strategy which is closer to the ‘optimal,’ as is increasingly being argued, would be to personalise choice architecture so that different people are nudged in ways which respect their individual differences (Lanzing 2019; Mills, 2020b; Peer et al. 2020; Porat and Strahilevitz 2014; Sunstein 2012, 2013; Yeung 2017).

At its core, this is a problem of selecting choice architectural designs dependent on the person being nudged (i.e., an input variable), and a problem which would seem resolvable by the introduction of machines (Sunstein 2013; Yeung 2017). Sunstein (2013) argues that the central technical challenge for personalised nudging is utilising heterogeneity data, or data which is required to determine differences between individuals (Mills 2020b). Additionally, Yeung (2017) argues that automated systems such as AI are valuable because they can respond much more rapidly to feedback than any human choice architect could; an argument which, essentially, is also a data analysis problem which emerges from sustained (i.e., intertemporal) data collection. Aonghusa and Michie (2021) have also argued that the amount of data which can now be drawn upon, and which is necessary to achieve precise and personalised choice architecture, is beyond reasonable human capacities to analyse. Therefore, they suggest AI systems may be better suited for data analysis within behavioural science. In short, autonomous choice architects could learn to be better selectors of choice architecture than human choice architects, could do this faster, and could resolve difficult data challenges which are currently beyond human capacities, with personalised nudging being the prime example of these advantages in combination.

To further illustrate how choice architecture may be arranged by an autonomous, AI system, we will briefly discuss how AI systems can and are acting as selectors of choice architecture in the areas of (a) policy evaluation; (b) ecommerce, and (c) information environments. We have chosen these areas for their historic prominence within the behavioural science literature (Sanders et al. 2018) and for their continued relevance within the contemporary digital economy. This discussion is given to be illustrative, not definitive or to develop any wider framework. For more complete frameworks regarding the interaction of behavioural science and technology for nudging, see for instance Mele et al. (2021), Schneider et al. (2018) and Villanova et al. (2021).

Our discussion has frequently drawn on contrasts between human choice architects and autonomous choice architects, but often only insofar as we argue that machines can replicate and surpass the functions of human choice architects. However, thus far a discussion of accountability has been missing. For instance, who is responsible for an algorithm which automatically architects choices to influence behaviour? As discussion of technology-linked nudging, such as hypernudging (Yeung 2017), Big Data nudging (Sætra 2019), digital nudging (Weinmann et al. 2016), smart nudging (Mele et al. 2021), and generally AI nudging, has developed, so too has the risk that human responsibility for the actions of these autonomous systems will slip out of focus. Some might intuitively see it as self-evident that using an AI system as a choice architect is akin to using any other tool, and that it entails no dissolution of human accountability. However, increased machine complexity and increasing machine autonomy has given rise to arguments implying that we must take seriously the possibility that machines that act intelligently can also be held accountable for the actions they take. We agree with those that intuitively hold that humans must be held accountable for the actions of machines, but merely assuming that this is self-evident leaves openings for opponents of such a view. To effectively argue in favour of human accountability in a world of autonomous choice architects, the preceding considerations about what AI is and is not capable of, and an understanding of how to bring this into an evaluation of potential shifts in accountability, is required.

As AI systems are used to either assist or replace human decision-makers, some argue that a “responsibility-gap” is thus created (Matthias 2004). This gap, it is argued, emerges because it is wrong, unfair, or otherwise problematic to attribute responsibility for machine actions to humans who cannot “anticipate, completely control, or answer for” these actions (Gunkel 2020). The problem is associated with the complexity and the autonomy of the machines. As machines become more complex, at some point complexity surpasses a human’s ability to understand the exact workings of the machine (Rahwan et al. 2019; Pedersen and Johansen 2020). In addition to complexity, certain systems use error as a method for learning and reconfiguration, and thus become inherently unpredictable (Matthias 2004). This is particularly relevant for unsupervised AI systems, where AI is tasked with identifying patterns and connections unknown to anyone, but remains an issue for supervised AI also, where understanding the process of learning can remain opaque (Pedersen and Johansen 2020). This produces the classic ‘black box’ problem of AI, where questions such as “what is the AI doing?” and “why is it doing that?” become extremely hard to answer (Durrell 2016; Pasquale 2015; Turkle 1988). Insofar as it is relevant to this discussion, the question, then, is: do complexity and unpredictability relieve humans of responsibility for entities such as autonomous choice architects?

We approach this question by considering how nudging can be automated and connected to AI systems. First of all, we argue that nudging is hardly ever direct in the sense that the choice architect directly interacts with the decision-maker. In changing the default on a form, or installing a poster in a cafeteria, the choice architect’s actions are expressed via the form, or via the poster. Thus, the choice architect is often directly involved in creating and modifying the choice architecture involved, but not in directly interacting with the decision-maker. From this perspective, the apparent indirectness arising from the autonomous choice architect is irrelevant to the question of responsibility.

Instead, a “veil of complexity” (Sætra 2021a) obscures the involvement of humans in automated and algorithmic nudging. In the first instance, AI systems constitute an additional layer of indirectness, as human choice architects are not directly involved in choice architecture and are not directly involved in applying it in settings in which the targets of nudges encounter them. Such ‘distance’ may be compounded when multiple choice architects are involved in designing the AI, making it harder to confidently determine who is responsible for what (Matthias 2004). In the second instance, a human choice architect may not be able to understand or explain the actions of the autonomous choice architect, and in a hypothetical confrontation between themselves and a decision-maker, may feel inclined to declare ignorance of the action and shirk responsibility. Yet, in lifting this veil of complexity, we argue that little actually changes in terms of who is responsible.

Classical examples of choice architecture manipulation accepted as nudges are the positioning of items in a store (near the check-out counter, for example; Thaler and Sunstein 2008), the positioning of items of restaurant menus (Dayan and Bar-Hillel 2011), and changing default options to encourage organ donation (Johnson and Goldstein 2004). All of these nudges can be performed manually, by a choice architect who selects how items are placed in the store, designs the menu, or changes the default option from opt-in to opt-out. This basic, or fully manual, form of nudging, is static and requires another manual action by a choice architect to change the nudge. Furthermore, all targets of the nudge encounter it in the same place, time, and manner if they behave similarly (Mills 2020b).

What, then, if the restaurant focused exclusively on app-based and individually delivered menus, and implemented a system for registering users and analysing information on these users with the use of intelligent systems? This could be done to maximally harness the potential of personalised choice architecture (Mills 2020b). For the sake of the argument, we assume that the restaurant is operated by a company that also runs many different types of stores and services, and that they already have a vast array of data points of each customer. Rather than manually deciding what sort of menu to show at a given time, or manually tailoring the menu to each person, the company decides to use unsupervised deep learning algorithms to decide how menus are presented. The ultimate goal is provided, to maximize long-term profits in this case, and the AI system provides the personalised menus in-app. No one, not the customer, and no one in the company or those developing the app or algorithm, (a) understands or can explain why individual X gets menu Y, and (b) no one has been directly involved in the particular choice architecture presented to the customers. This, one might argue, is how the autonomous choice architect is born.

The latter form of system involves distancing the human choice architect from the actual end results, but we argue that despite the unpredictability and opaqueness of such systems, it is imperative to recognise humans’ role in the process as choice architects. While sophisticated deep learning systems introduce a veil of complexity, a proper understanding of these systems, both of the algorithms and the data, allow us to remove this veil. We emphasise five keyways in which human influence remains clear. First, humans program the systems and algorithms involved, and this will always entail making a range of choices regarding how the systems will end up making decisions. Secondly, humans decide how, where, and when these systems are to be applied, such as in a particular restaurant. Thirdly, humans directly instruct these systems to optimise based on selected variables. Fourthly, humans are involved in deciding which variables or factors, which parts of the choice architecture, are manipulable by the AI system. Fifthly, and finally, humans are involved in a wide range of actions that shape and influence the generation, selection, and codification of the data used by these systems (Sætra 2018).

Humans are irrevocably involved, even when autonomous choice architects are used. Granted, the involvement is several steps removed, and appears more indirect, and may be orientated more to the design of a machine than the design of choice environments, but insofar as responsibility is a function of control (Gunkel 2020; Matthias 2004), and insofar as control over an autonomous choice architect can be attributed to whoever sets the parameters of it, humans remain responsible (Sætra 2021a). If machines have become autonomous choice architects, this is, if anything, delegated or automated choice architecture, and human beings remain responsible.

Nudges are supposed to allow individuals to “go their own way” (Thaler and Sunstein 2008, p. 5). They are, in a sense, suggestions which a decision-maker should be able to ignore (Madrian and Shea 2001), even if they are suggestions designed to exploit human bias. Much objection has already been made towards what we have here called manual nudging (Rebonato 2014; Mitchell 2005), and we do not wish to restate these arguments, particularly in regard to freedom of choice. Yet, the very nature of autonomous choice architects, owing to their mechanical motive power, mean they are able to respond faster to decision-maker feedback, able to integrate a more perfect image of the decision-maker in the design selection, and inevitably able to attune their selections to a specific decision-maker in real-time (Sætra 2019; Yeung 2017). Automatic nudging, therefore, may only nominally accord to nudging insofar as it allows one to “go their own way,” as automatic nudging can design choice architecture, collect feedback, and change choice architecture so quickly, and so efficiently, that a decision-maker may never be able to escape its influence.⁸ Insofar as such systems continue to be described as ‘nudges,’ while not substantially allowing people to ‘go their own way,’ autonomous choice architecture may present a significant ethical challenge for the wider nudging programme.⁹

The consequence of this is it compounds the veil of complexity by allowing a choice architect to excuse their influence on the basis that decision-makers can always ignore them. If a decision-maker is dissatisfied, so the argument might go, it is their own fault for following the nudge. Of course, if one’s ability to not follow is compromised, such an excuse is not valid, but could still be made in an attempt to avoid responsibility.

Finally, a long-standing criticism of what we have called manual nudging is that there is no particular reason to believe that a choice architect should know any better or any worse than a decision-maker regarding what the ‘best’ or ‘correct’ choice is (Selinger and Whyte 2010). This is largely because a) people are often different, with different preferences and motivations, and b) the choice architect is often removed from those they are architecting choices for, and may even be biased themselves (Rebonato 2014). In some ways, autonomous choice architects may function as a response to this otherwise tricky criticism of nudging by utilising individual-level data to personalise choice environments (Mills 2020b; Porat and Strahilevitz 2014; Yeung 2017; Mele et al. 2021).

However, autonomous choice architects may compound the problems of knowledge, competence and trust discussed by Selinger and Whyte (2010) by robbing human choice architects of any semblance of the possibility of explaining to a decision-maker why they were nudged in the manner that they were. Insofar as we argue human choice architects are ultimately responsible and accountable for the actions of autonomous choice architects, autonomous choice architects may, rather than empowering humans, act as an impediment to effective discourse between choice architect and decision-maker about why and how nudges should be used.

Autonomous choice architects already exist, and insofar as they allow many services to be automated and to convey social benefit via, say, tailored information, disclosures, or default options, may also be described perhaps more positively than we have done in this article (see Mele et al. 2021). Yet many of these benefits-speed of reconfiguration, ability to integrate individual-level data, responsiveness to feedback in real-time-produce implications which prompt reconsideration of the implications of the role of the choice architect.

From the practitioner’s perspective, the autonomous choice architect may be seen as a threat, not so much from the perspective of automating away their job-research into behavioural science and nudge design seems likely to continue to be a human-dominated domain for the moment-but from the shifting responsibility they bring. Practitioners need to understand the systems they are implementing, not only on a technical level (insofar as this is possible; Burrell 2016), but also in terms of their responsibility over what the system does and why. We can break these questions down further into more specific inquiries:

These questions likely vary from context to context (e.g., a private-sector choice architect may have different obligations to stakeholders compared to a public-sector choice architect; Beggs 2016), but we consider them a worthwhile initial set on questions for the behavioural (and data) science community to begin with.

In terms of implications for society more broadly, recent developments show that regulators are adopting a stance that might drastically limit uncritical choice architecture automation. The European Union’s GDPR framework already introduced a right to explanation, or at least a right to meaningful information about the logic involved in automated decision making (Selbst and Powles 2018). More recently, however, the European Commission has proposed a new AI regulation that goes further in limiting how AI can be used for changing people’s behaviour (European Commission 2021). Here they propose a ban of AI systems thought to entail “unacceptable risk”, and one of the examples mentioned is “systems or applications that manipulate human behaviour to circumvent users’ free will”. Whether or not autonomous choice architects manipulate or not is a contested question (Sætra 2021b), but the phrasing in the proposal is vague enough for the systems we discuss in this article to potentially be included. In addition, the proposal suggests that certain contexts are “high-risk”, such as education, critical infrastructure, and law enforcement. They proceed to suggest that any use of AI in such contexts should be associated with strict requirements related to, for example, “appropriate human oversight”, “adequate risk assessment and mitigation systems”, “clear and adequate information to the user”, “high quality of the datasets,” and a “high level of robustness, security and accuracy” (European Commission 2021). Regardless of the outcome, it seems clear that governments are increasingly taking an interest in mitigating potentially negative consequences of AI. This is both a consequence of increasing awareness of the implications associated with particular technologies, such as the ones discussed in this article, but also to a potential shift in the willingness to allow the political domain to limit the relatively free application of technological innovations (Sætra and Fosch-Villaronga, 2021).