The problem with trust: on the discursive commodification of trust in AI

Trust is all the rage in thinking about artificial intelligence (AI) and society. It dominates international principles and guidelines developed by the private sector (IBM 2018) and by multilateral organizations (OECD 2019). It takes center stage in national strategies for AI (Misuraca and Van Noordt 2020) and in the mandates for proposed national regulatory institutions (Mulgan 2016). The idea of trust grounds scholarly proposals for AI governance regimes (Janssen et al. 2020) and experimental tinkering in the programing of algorithms (Ribeiro et al. 2016; Toreini et al. 2020). It frames civil society calls for ethical AI (Balaram et al. 2018), is extolled as “essential” by social commentators (Larsson et al. 2019), and spurs the launch of boutique consultancies.¹ Indeed, a recent mapping of efforts to strengthen the governance of AI finds a proliferation of principles “from all sectors, including governments, corporations, and civil society,” and notes that the “stated purpose of many of the principles is to forge trust: between governments and citizens, between corporations and consumers or users, and between AI researchers and the general public” (Cussins Newman 2020, p. 11).

Critical voices in this discourse are few and far between. After all, who would want to argue against the importance of trust and trustworthiness? Yet, some stakeholders have pointed towards the undue strain that is being put on the concept. A disillusioned member of the High-Level Expert Group responsible for developing EU ethics guidelines, for example, criticizes the notion of trustworthy AI as “conceptual nonsense”:

Machines are not trustworthy; only humans can be trustworthy (or untrustworthy). If, in the future, an untrustworthy corporation or government behaves unethically and possesses good, robust AI technology, this will enable more effective unethical behaviour (Metzinger 2019).

Warnings such as Metzinger’s seem to go mostly unheard in a discourse that is dominated by enthusiasm for the societal and market benefits assumed to follow from AI technologies. Instead, the literature considered in this commentary suggests that the concept of trust has a near-universal utility. Trust in AI is imagined to help governments gain legitimacy (Freuler and Iglesias 2018; Kuziemski and Misuraca 2020); to improve customer satisfaction with services (Al-Mushayt 2019; Berscheid and Roewer-Despres 2019; Matheny et al. 2019; Mehr 2017); help societies realize their potential for social good (Mikhaylov et al. 2018; Tomašev et al. 2020); and unlock market value (Chakravorti and Shankar 2017; Gordon Myers and Nejkov 2020; Orr and Davis 2020). Simultaneously, trust is widely applied as a quality seal for “good AI”, and often collocated with “excellence”, or “cutting-edge” technology, as in The European Commission’s “White Paper on artificial intelligence: a European approach to excellence and trust” (European Commission, 2020, our emphasis).

A complementary strain of literature, meanwhile, warns of the risk that AI can fail to achieve its objectives when it is not trusted. Villani (2018), for example, states that “widespread distrust of AI on the part of the general public, […] in the long run is liable to curb its development and all the benefits it could bring” (116). Just as social media platforms wither away when they are not being fed with the interactions of a critical mass of people, AI will not develop sufficient intelligence—will remain dumb and unreliable—when left unused. Such “bad AI” can then lead into a vicious circle in which already lacking public trust is damaged further by the insufficient number of interactions it receives (Reisman et al. 2018).

These hopes and fears must be understood in light of the market forces which drive them. The emergence of AI technologies corresponds with the emergence of new markets, as indicated by stiff competition between private companies, and efforts by governments to strengthen national competitiveness (European Strategy for Data 2020; Misuraca and Van Noordt 2020). The coupling of strong incentives to advance AI systems with the inherent uncertainty surrounding AI’s societal impact then leads to a polarized dynamic in which a skeptical public (Anderson et al. 2018) is matched by “AI cheerleaders” working to normalize socio-technical systems on behalf of private interests (Bourne 2019). The risk that this polarization can slide into paternalistic models of technological governance is very high indeed (Cardullo and Kitchin 2019; Oravec 2019) and should be taken seriously. As Cath (2018) notes when considering AI governance regimes

…industry efforts [to develop best practices] are laudable, but it is important to position them in light of three important questions. First, who sets the agenda for AI governance? Second, what cultural logic is instantiated by that agenda and, third, who benefits from it? Answering these questions is important because it highlights the risks of letting industry drive the agenda and reveals blind spots in current research efforts (Cath 2018, pp. 3–4).

It is in this vein that we question the prominence of trust in the contemporary discourse on AI and society. To be clear, we do not assert that trust is in any way unimportant. Efforts that address widely held public concerns about AI’s societal impacts are laudable, including efforts to build trust. We are concerned, however, that this near-universal attention to the concept obscures exactly the challenges that make trust in AI problematic in the first place.

This analysis is not an exploration of trust in AI or its conditions, whether they are present or ought to be, or whether trust is commodified in an economic sense of specific user interactions or market dynamics. Indeed, we are not concerned with trust per se, but the way in which the notion of trust is asserted and leveraged within a very specific discourse. This commentary is concerned with the commodification of trust as an important aspect, or moment, in the public discourse on AI and society, and the implications that moment might have for how AI is understood and implemented. Specifically, we are concerned by the increasing practical and instrumental attention to the general public’s trust in AI as a precondition for generating the vast quantities of data required by machine-learning systems. Hence, trust becomes asserted as something that should be provided blindly, as a resource to be extracted, and as an instrument for unlocking value. Under these circumstances, the discursive commodification of trust is closely associated with the eagerness of private and government stakeholders to promote the implementation of AI systems in the name of trust, while turning a blind eye to the question of whether those systems are actually deserving of people’s trust.

Our argument proceeds with a brief description of the theoretical background and methods that inform this commentary. This is followed by a third section outlining the relevant conceptions and levels of functioning of trust in the context of AI, before addressing the problems that can be associated with the existing perspectives. The fifth section concludes with an assessment of what trust has come to mean in the contemporary discourse and offers critical suggestions about how to counter the biases that this discourse produces to safeguard the common good and public well-being.

Methodologically, what follows is best understood as a critical discourse analysis aligned with the understanding of socio-linguistic conventions as the expression of power relationships and power struggles (Fairclough 2010). In our application of the method, we depart from Michel Foucault’s (1972) definition of discourse as relations between statements, groups of statements, and other events, even if these statements and events have no prior connection to one another. In contradistinction to Foucault, however, we make room for the social effectiveness of non-discursive spheres in line with Norman Fairclough (1995) as well as the importance of a notion of subjectivity irreducible to social and discursive positions (e.g., Henriques et al. 1984; Carolan and Bell 2003). We believe that this is a productive approach to exploring the ways in which ideologies and hegemonic values are asserted and accepted in the discourses surrounding new technologies.

This approach is generally aligned with the tradition of critical social theory that understands these dynamics in terms of power and power relationships between groups and individuals (see Stoddart 2007). While the power relationships at play in this discourse are not the main focus of our analysis, this framing is particularly important in the context of AI’s simultaneous opacity and ubiquity. We hope that our analysis can guide attention to the powerful interests that are driving the current discourse on AI governance, or to ask in Cath’s (2018) language, who is setting the “agenda.” This approach is also well suited to an analysis of the role of trust in such a discourse. Not only is trust widely recognized to be a discursive phenomenon, constructed and situated within social systems (Carolan and Bell 2003), but a critical discourse theory provides a productive framework for understanding how notions of trust weigh in on and influence governance paradigms, precisely because “trust-related and trust-bearing issues are central to our understanding of how the conduct of professional practices impacts on human relationships in social life” (Candlin and Chrichton 2013, p. 1). In this respect, our approach departs from the analysis often applied to the micro-level of communications in Information Systems research (Cukier et al. 2009), but remains focused at the macro-level of public discourse and how the social reality inscribed in it is indicative of the balances and imbalances of power in society. This also follows the theoretical orientation of critical policy studies to understand that the language used in public documents with a scholarly or policy orientation will play a key role in staking out what is politically possible and desirable in regard to AI governance as a policy problem (Mulderrig et al. 2019).

Furthermore, our theoretical orientation is reflected in our use of the term “commodification”. There is a significant body of economic research conceptualizing trust as a commodity, which can be traced from Zucker’s (1986) seminal work on the production of trust in macro-economic structures, to contemporary game-theoretical work on the commodification of trust in specific economic interactions (Dasgupta 1989). Recently, there has also been increased interest in the commodification of trust in regard to new technologies, and Bodø (2021) for example explores how blockchain technologies “transform trust and trustworthiness, a form of social capital, into a commodity, an industrially produced asset that can be quantified, traded, enclosed, and sanctioned” (p. 2678–2679). Our critical discourse analysis departs from this economic approach in favor of a Marxist understanding of commodification and a critique of neoliberal practices that leverage new technologies to assert and reinforce power relationships (Cardullo and Kitchin 2019). Unlike comparable critical readings, however (e.g., Arvanitakis 2007), we are attuned to this process at the level of discourse, rather than at that of individual’s relationships or social realities.

Our sample for assessing the academic and policy discourse on AI and society is selective, and does not aspire to be strictly representative, inviting a critique that is often levied at critical discourse analysis (Breeze 2011). We nevertheless believe that a more partial review of literature is justified given the highly diffuse and fragmented nature of the discourse (Cussins Newman 2020) and the imprecise nature of trust as a concept. Our sample is nonetheless broad, composed of over 100 publications addressing AI, society, and trust. The sample includes academic journal articles (n = 65), government strategies and whitepapers (n = 29), analyses from Non-Governmental Organizations (NGOs) and think tanks (n = 10), reports from industry consultancies (n = 2), and journalistic opinion pieces and blog posts (n = 5).

For all its commonsensical appeal, trust is a surprisingly slippery concept, and as Hardin (2002) notes, there is “no Platonically essential notion of trust. Ordinary-language usages of the term trust are manifold and ill articulated” (p. xx). This ambiguity makes sense to a degree, if one understands trust as a social phenomenon, discursively constructed, bound by context, and continuously negotiated as a basis for social interaction (Candlin and Crichton 2013, p. 9). Ambiguity in this context is also what makes trust such a powerful social phenomenon, as a fundamental condition for complex social action and cooperation (“Trust Mak. Break. Coop. Relations” 1989), and underpinning the social construction of truth itself (Carolan and Bell 2003).

A common language understanding of trust will often emphasize the notion of trust cultivated through a process of learning from experience in interpersonal relations over time, and has been the focus of a significant strain of psychological research (e.g.: Berzoff 2011; Erikson 1963). This type of trust, situated in individual interactions and expectations, has been integrated with sociological conceptions of “system trust” in which trust is marked by increasing complexity and challenges to anticipating the behaviour of others, including complex social systems (Giddens 1990; Granovetter 1985; Luhmann 2017). Though sociological research has increasingly focused on the de-personalized character of trust in such complex systems (Shapiro 1987), there is widespread agreement among scholars that individuals’ trust—be it in other people, institutions, or systems—is always a matter of both prediction and normativity and is “always associated with expectations about the behaviour of others” (Candlin and Chrichton 2013, p. 2). This resonates with Hamm et al.'s (2016) cross-disciplinary review of what they call institutional trust, which emphasizes perceptions that institutions are protecting people’s welfare as the primary antecedent of this type of trust.

For this commentary, it is not necessary to trace the messy debate about micro- and macro-level dynamics that contribute to building and eroding trust (e.g., Bachmann and Inkpen 2011). Indeed, there is every reason to believe that different types of trust are operating simultaneously, as has been demonstrated in experimental research on individuals’ trust in online financial interactions (Pennington et al. 2003). To explore how the notion of trust is manifest in the discourse on AI and society, it is nevertheless useful to note that there are several ways in which trust might be cultivated in novel technologies like AI. While, in keeping with Hamm et al. (2016) and critics like Bodó (2021), trust might be earned with the exhibition of trustworthy behaviour, other studies indicate that, under certain conditions, people are willing to trust in emerging technologies even if these technologies have by no means earned their trust (e.g., Mazey and Wingreen 2017).

Furthermore, dynamics of trust in AI occur in imperfect information systems, and complex systems introduce multiple vectors for mistrust (Shapiro 1987). This has led to several proposals for how trust can be created and vested through proxy, including notions of trust mediators (Bodó 2021), data stewards (Janssen et al. 2020), or algorithmic social contracts to embed societal values in AI design (Rahwan 2017). Other research has emphasized the importance of institutional mechanisms to foster trust and mutual accountability between the different sectors involved in AI development (Brundage et al. 2020), and suggests that it might also be necessary to introduce macro-level systemic changes to cultivate trust at scale. This aligns with Steedman et al.’s (2020) exploration of trust in data-driven systems, “particularized solutions to generalized problems are unlikely to be effective. We need collective, ecosystem solutions, for example, better regulation of data-driven systems, in order for them to be perceived as more trustworthy” (p. 829).

Because trust is a matter of perception and predication, however, trust might also be cultivated without the performance of trustworthy behaviour. It is in this vein that Bourne and Edwards (2012) describe the discursive strategies deployed to cultivate system trust in the hedge fund industry. In the same vein, critics of AI systems lament the emphasis on public relations campaigns and marketing that attend to AI technologies and technology vendors (Bourne 2019; Cardullo and Kitchin 2019; Oravec 2019). For the present commentary and analysis, then the question becomes which of these responses is most prominent in the contemporary scholarly and policy discourse on AI and society.

What becomes apparent from the discussion so far is that it confronts researchers and stakeholders with a challenge: To a significant degree, the object one is asked to invest one’s trust in is still in the process of emergence. The very notion of AI is under public construction, both in the expert discourse reviewed here, and in the experiences and perspectives of the individuals that become AI users. Shifts and changes in sociocultural meanings affect the conditions for trust, and the global discourse sets the stage for how (and if) those conditions will be manifest. It is from this constructionist vantage point, which conceives of objects as being shaped by the ways in which they are understood, that we want to draw critical attention to the meanings that we see as emerging from the recent literature, where trust is frequently approximated to blind trust, to a resource, or an instrument.

Our critical analysis of the discourse on AI and society suggests that there is an increasing trend towards the discursive commodification of trust. We find an overwhelming emphasis on the conditions for developing AI systems and unlocking markets, which conflates the interests and agendas of AI users, experts, and regulators, with those of technology vendors, venture capitalists, and financial backers, whose interests in advancing AI might be diametrically opposed. More critically, this emphasis has the effect of obfuscating the conditions under which trust between individuals and AI systems is developed and earned.

The insistence that governments focus on building public trust in AI systems obscures the questions about what might be required to actually build trustworthy systems. When trust is commodified in this discourse, there is no space to ask whether macro- or micro-level approaches are more appropriate, whether trust is justified, or who or what precisely users are asked to trust in. This might be dismissed as rhetorical emphasis; we hold, by contrast, that it is of critical importance, because the policy and scholarly discourse on AI and society is actively staking out the policy options for AI governance, and those governance structures will have far reaching and profound implications for society (Cath 2018; Cussins Newman 2020; Mulderrig et al. 2019). When appeals to build trust fail to acknowledge the intricacies of how trust gets built, they make the notion evermore elusive for both the designers of AI systems and the users.

Countering this discourse’s gravitation toward blind, commodified, and instrumental trust, we would encourage experts, politicians, and stakeholders to instead focus on the work and the investments that indeed remain preconditional of the public’s trust. Though not prominent in the discourse, there is solid research on which to base such an effort. Trustworthy AI may well require programming and design that is technologically robust (Bellamy et al. 2019; Etzioni and Etzioni 2016; Harrison et al. 2019; Kroll et al. 2017; Liao and Muller 2019; Sokol et al. 2020; Veale et al. 2018), processes and policy frameworks that protect citizens in the slightest case of doubt (Balaram et al. 2018; Brundage et al. 2020; Dignum 2019; Janssen and Kuk 2016; Kemper and Kolkman 2019; Kolkman 2020; Lee 2018; Mulgan 2016; Reisman et al. 2018; Vassilakopoulou 2020), or public-sector routines and protocols that allow for AI and human-service offers to run side-by-side for the foreseeable future, questioning, and learning from each other (Berscheid and Roewer-Despres 2019; Janssen et al. 2020; Katell et al. 2020; Rahwan 2017; Vestby and Vestby 2019; Yeung and Lodge 2019).

These might not be necessary mechanisms for building trust in AI, or the most appropriate or effective. But until the scholarly and policy discourse assessed here becomes actively engaged in an exploration of which mechanisms are the right mechanisms for building trust, the discourse on trust in AI is a shell game, baiting with market value and technological glitz, ignoring the hard questions about why trust matters. With the current discourse, governments and businesses are in no position to approach citizens and customers with a plea for their trust in shiny new systems. As Dignum (2019) writes: “Ensuring ethically aligned AI systems requires more than designing systems whose result can be trusted. It is about the way we design them, why we design them, and who is involved in designing them” (v). We would add that it is also about fostering a discourse that actively acknowledges and engages with how this should be done.