Objections to Simpson’s argument in ‘Robots, Trust and War’

Simpson does not justify his claim that trust is most commonly based on the good character of the trustee, and I argue that this is unlikely in general because it would be challenging for a trustee to be of good character, or for a trustor to recognise them as such. I support my claim with examples that show that this claim is also unlikely to hold true in the particular context of counter-insurgency engagements.

That is, to be trustworthy based on good character according to Simpson’s account, a trustee needs to consider how to behave, taking account of the trust placed in them by the trustor as just one factor. This is quite demanding as it requires the trustee to be capable of both moral reasoning and of consistently acting on those reasons.

Furthermore, for that trust to be rational as Simpson presumes it should be, he claims that it needs to be based on good evidence that it is likely to be well-placed. However, as Hardin (2002) has already argued, getting evidence of good character is challenging as it would require significant familiarity with the trustee. To elaborate on this, it would require not only evidence of a potential trustee’s actions in a variety of contexts, and over time but also an appreciation of the motivation behind them—because only some types of motivation would be consistent with good character. For example, a trustee would not usually be regarded as being of good character if they complied with a trustor’s expectations merely for reasons of self-interest.

Evidence of a trustee’s real motives are also likely to be concealed where there are significant levels of external controls over their actions (Wright 2010). For example, in the particular context of a counter-insurgency force, the governing body specifies objectives and there are rules and values that soldiers are expected to comply with. Their behaviour is scrutinised by colleagues, their command, other participants and civilians, and those deviating from what is required are likely to be subject to punishment. As a consequence, it could be rational for civilians to trust individual soldiers on the basis of their self-interest as it would appear to be in their interests to conform.⁴ However, this would make good character difficult for civilians to discern as it would not be clear whether behaviours were driven by the good character of a soldier or these external controls. Trust based on the good character of soldiers would therefore be unlikely, even if there was the potential for it.

Simpson (2013) has disputed Hardin’s claim that getting evidence of good character is challenging on the grounds that reliance can be placed instead on the judgement of others, or of institutions to which the potential trustee belongs. In the context of counter-insurgency engagements presumably reliance might be placed on participation in bodies such as the UN and NATO, and commitment to the Geneva Convention and an ethical foreign policy. However, I argue that this begs the question because the trustor would still need to satisfy themselves that those individuals and/or institutions could in turn be trusted to provide such broad-ranging assurance. Most likely they would provide assurance in only limited respects, so multiple sources would have to be sought and the trustor would have to seek sufficient grounds for trusting each. Furthermore, some institutions may establish only principles to guide behaviour that are subject to interpretation, and therefore may not provide the level of assurance that is required. In the circumstances of counter-insurgency engagements finding a sufficient basis for such assurance may be particularly challenging, and essentially impractical for civilians.

In the same paper Simpson also argues that ‘a decent moral character may be permissibly assumed in situations where it does not matter too much if the assessment is wrong and decency is sufficiently prevalent across a population’ (2013 p554). However, in the particular context of a counter-insurgency war it is unlikely that either condition is met. Not only are the very lives of civilians often at stake, but counter-insurgency forces may not even be recognised as part of the civilian population—for example if they are foreign or robotic—making it implausible that civilians would assume that they had a level of moral decency that matched their own.

Therefore, contrary to Simpson’s claim, normative trust based on the good character of the trustee seems unlikely to be the most widespread form of trust—both in general and in the particular context of a counter-insurgency engagement—because it is more difficult for a trustee to achieve, or for a trustor to establish. I next argue that this form of trust is also unlikely to be appropriate in the context of counter-insurgency engagements.

I claim that interpersonal normative trust based on the good character of the trustee is also unlikely to apply in the particular context of a counter-insurgency engagement because it is not feasible, necessary, or desirable. I argue that it is not feasible because the nature of these conflicts means that it is questionable whether soldiers in counterinsurgency forces could ever be perceived as being of good character by civilians because of significant mismatches between the values of the respective cultures. For example, Egnell (2010) claims that ‘traditional sources of legitimacy in Afghanistan, (are) often based on identity and cultural affinity’, and that the international coalition are foreigners with values that are ‘oceans apart’ from local civilians (p299). It is therefore not simply that counter-insurgency forces could not be rationally trusted because they are unable to adhere to moral values, it is that even if they did so, those moral values could be very different to those of the civilians. This is consistent with an observation made by Lewicki et al. (2006) that initial distrust between parties may arise from cultural/social factors, an untrustworthy reputation or other contextual factors, which Kramer (1999) has suggested may form psychological barriers to the formation of trust. Efforts to win civilian ‘hearts and minds’ based on making robot counter-insurgency forces trustworthy—as measured by the values of their developers—may therefore ultimately be misguided.

It might be argued that Simpson’s account of good character is sufficiently thin to allow overlap between civilians and counter-insurgency forces.⁵ However, I claim that even if this is the case, areas of divergence could plausibly prove more influential than areas of overlap because trust is usually characterised as being easier to lose than to gain.

Furthermore, if Simpson’s account of good character is taken to be very thin then it will be insufficient to support his other claims i.e. that it is the basis for winning ‘civilian hearts and minds’ and excludes robots from being trusted. That is, if good character is interpreted in the thinnest sense that Simpson describes as ‘having the stable disposition to think certain things to be important and valuable, and being competent to act sensibly in light of these’ (p328) then robots could be capable of it already because they can be programmed with some limited set of values and are arguably competent to act ‘sensibly’—depending on how this is interpreted. For example, they might be considered to act ‘sensibly’ on such values under most circumstances if they could at least prioritise them—even if unable to do so in a more sophisticated way e.g. by resolving conflicts. Furthermore, winning ‘hearts and minds’ requires that soldiers take the interests of civilians into account—so they would need to respect not just any values, but those particular values that are pertinent to civilians. If they did not then their actions would not provide sufficient evidence of good character in civilian terms for them to be rationally trusted.

Even if certain values are the subject of ‘invariant agreement’ as Simpson suggests (p329) and are therefore shared, their interpretation and/or application may still differ between civilians and soldiers resulting in a loss of trust—and this may be exacerbated by membership of the counter-insurgency force. By way of example, it is likely that there will always be tension between the demands of soldiers as members of a counter-insurgency force and the requirement to be of good character as a civilian. Both civilians and forces may value integrity, duty, and loyalty—but as soldiers Simpson observes that “the most fundamental and deeply held value is the successful completion of the given task” (p326). As a consequence, other values which may be more important both to civilians, and even to soldiers as individuals, may be given a lower priority when ensuring that a particular end is achieved.

It may also be unnecessary for robot forces to win hearts and minds because they may not materially contribute to achieving this objective for two reasons. Firstly, because perceptions of the force as a whole may be more influential in establishing trust than the actions of individual soldiers.⁶ This is because soldiers are not autonomous in many ways as they are subject to the rules of the force (i.e. including its governing body) who define the objectives of the engagement, as well as the rules and values that should be adopted in pursuing them. Civilians therefore remain vulnerable to betrayal by the choices of another as Simpson’s account of trust specifies—but this choice is largely in the hands of the force and not individual soldiers.

Secondly, there are alternative interpretations of, and approaches to, achieving ‘hearts and minds’ which involve different tactical activities in which military forces do not, and perhaps should not participate. That is, Simpson’s account claims that soldiers would need to ‘simultaneously conciliate and build relational bridges at the same time as defending themselves’ as the basis for engendering civilian trust (p 331)—which suggests an assumption of a generally less coercive approach to an engagement. However, Egnell (2010) claims two alternative approaches involve undertaking humanitarian and development activities and specialist information operations. He suggests that these may not require, and possibly should not involve military forces at all because it could undermine civilian trust in them. For example, if military forces are less well equipped than others to undertake humanitarian activities then they are unlikely to be trusted because not only might they fail to deliver as promised, but their involvement may confuse civilian expectations of them.

It might be argued that it would still be necessary for soldiers to be trustworthy—or at least not untrustworthy—whatever their involvement in a counter-insurgency engagement and however extensive their role because distrust between civilians and soldiers on the ground could stymie attempts to create trust through these other means. However, I claim that this could be achieved without requiring them to be trustworthy based on good character⁷ as I explain in the following section, and it may even be sufficient that they are merely trusted predictively.

Finally, I argue that good character on Simpson’s account may not even be desirable in counter-insurgency forces because it may not support what civilians and counter-insurgency leadership require them to do. As Wright (2010) has argued, if trustworthiness is a moral characteristic, then it does not necessarily entail doing as a trustor expects. A trustee acting on the basis of good character may take a trustor into account yet decide not to do as they expect if they consider the action morally wrong—or less morally valuable than some other action external to that relationship. The good character of a trustee is therefore only a good reason for a trustor to trust them if what they are trusted to do is something that the trustee is also likely to regard as morally correct. For example, a civilian might rely on counter-insurgency forces to shoot insurgents on sight if that is what they expected of a protective force. However, if the force was trustworthy in a moral sense, it might hesitate to kill insurgents rather than capture them. For similar reasons it may be undesirable for the leadership of counter-insurgency forces to deploy forces that are trustworthy based on good character if this leads to them questioning the morality of the war, its conduct and operations.

I note however that even if trust based on good character is neither the most common form, nor what is required of counter-insurgency forces, it may still be the most important in general—particularly if it helps to maintain trust in society as a whole (e.g. McGeer and Pettit 2017) though consideration of this matter is beyond the scope of this paper.

In conclusion, I have argued that rational, normative trust based on the good character of the trustee is unlikely to be a common form of trust, and is not feasible, necessary or even desirable in the context of a counter-insurgency force. In the following section I suggest alternative grounds for rational normative trust in soldiers that are arguably more plausible.

As Simpson notes, rational normative trust based on the good character of the trustee provides assurance about their behaviour even in circumstances where what is expected of them cannot be specified in advance. Although common-sense dictates that this would usually be desirable, I argue that it may be sufficient for a trustee to be trusted on more limited grounds—and that this is likely to be more common because it is less demanding.

For example, Cogley (2012) suggests alternative grounds for trust which seem particularly plausible as they explain feelings of betrayal—which is widely acknowledged to be a core feature of trust. Rather than grounding trust entirely in features of the trustee i.e. on their good character, he claims instead that trust may involve the trustor feeling entitled to the consideration of a trustee because they are party to a normatively characterised relationship with them. The trustor would feel betrayed if the trustee breached their trust in this case because not only would they lose the expected benefits of that trust, but also because it would adversely affect the relationship between them. On such grounds the trustee would therefore be trusted only within the particular domain of that relationship, and this trust would be rational if the trustor had evidence that the trustee was likely to conform with their, and perhaps societal expectations of that role.⁸ On such grounds the trustee could even be trusted to do something that was immoral if that was what was expected within the scope of that relationship. For example, a criminal could trust a colleague to help rob a bank.

By way of example, I could trust my regular mechanic to fix my car–not because he is trustworthy as matter of good character but because he is competent to do as I expect, and has good reason to fulfil his commitment to me as a tradesman. This is not simply predictive trust as it has the characteristics of interactivity that Simpson claims distinguish predictive from normative trust as set out in my summary of Simpson's account of trust i.e. I trust the mechanic because I rely on him to fix my car, he is aware of this and we both believe that he ought to take my trust into account because we have an implied relationship in virtue of his role. If he did a bad job nonetheless then I would feel betrayed because not only would my car not have been mended but our relationship would also have been undermined.

I claim that this form of trust addresses the two practical difficulties of trust based on good character that I identified in the previous section, and is therefore plausibly a more common form of rational, normative interpersonal trust. That is, it is less demanding of the trustee as they need only to conform to the trustor’s expectations of them. Furthermore, the trustor would require less, and more easily available evidence for this trust to be rational. This could include anything that increased the likelihood that the trustee would behave as they ought to, but only within the specific domain of the relationship. It would not even necessarily require evidence of trustee motivation—unless this was constitutive of that relationship. For example, caring for the other is usually constitutive of a close friendship—so in this case, there would need to be evidence that the friend was motivated to look after you in various contexts simply because they were your friend and not, for example, because they believed that it was in their self-interest to do so. However, in less demanding relationships such as the mechanic example, I may trust them regardless of why they are fulfilling their role—as long as there is evidence that they are likely to continue to do so.

This account also better explains how in practice we typically trust others to behave in a certain way in some circumstances but not in others, and about some things but not others—as this is dependent only on the scope and nature of the trustor’s relationship with the one who is trusted. For example, I may trust a friend to be honest with me without any expectation that he will also tell his mother the truth. In contrast, Simpson’s account implies that to be trusted on the basis of ‘good character’ would require the behaviour of a trustee to be consistent on a much broader basis—though I acknowledge that the extent and nature of this is likely to depend on exactly how his account of the ‘good’ in ‘good character’ is interpreted. A consideration of the possible alternatives is beyond the scope of this paper however.

The immediately obvious objection to applying these grounds for trust in respect of robot forces is that trusting robots would arguably be little more than reliance (or predictive trust) given that relationships between robots and civilians are not properly interactive, and we do not usually feel betrayed by technology when it fails to perform. This is only the current state of affairs however, and in the future, this will depend on how our relationships with robots, and their capacities, develop. If robot functionality remains limited then they may continue to be considered simply as tools and trusted predictively. In this case only their originators, rather than the robots themselves, would be thought to be responsible and accountable for their actions. On the other hand, if they became more functionally advanced civilians might experience some sort of interactive relationship with them and might indeed trust them and feel betrayed in the event that their trust proved misplaced.⁹ This is not as outlandish as it seems, for reasons that I address in the section below.

There are therefore alternative grounds for rational normative trust that are less demanding of both trustees and trustors, and are consequently plausibly more common as the basis for civilian trust in robot forces. In the following sections I argue, also contra-Simpson, that not only are motives unnecessary for rational normative trust, but that robots might even be trusted on this basis without having any particular capabilities at all.

According to Simpson’s account, robot forces could not be rationally trusted because they cannot act from a motive based on good character. They would instead display failures of judgement because their ‘rule-following behaviour’¹⁰ would lead to action with perverse outcomes’ (p332). In this section I argue that trust based on the motives of a trustee is not necessarily rational, and that motives may not even be necessary for values-based behaviours. Furthermore, robots might still be rationally trusted even when there are ‘perverse outcomes’ due to rule-following behaviour.

That is, Simpson claims that robots could not be rationally trusted without having a motive to act. Convention is that normative and/or moral motivation is based on the reasons that an individual has to act (Rosati 2016), although the relationship between moral/normative reasons and motivation is not well understood. Whilst some believe that reasons are directly motivating, others require that both reasons and desires are involved. If the latter is the case then arguably it would be irrational to trust robots if they could act from a motive—because just like humans their own desires might then overwhelm whatever reasons they had to act.

By maintaining that motives are necessary for trust, Simpson also assumes that the processes involved in values-based reasoning and decision-making must necessarily operate in the same way for both robots and humans. It is not obvious why this should be the case however—and values-based behaviours could perhaps be achieved in robots through other means whilst still supporting the kind of trust that I describe in the previous section.

Even with the possibility of perverse outcomes due to rule-following behaviour I claim that it is not obvious that it would be irrational to trust robots however, because the occurrence and severity of these may eventually be reduced sufficiently for trust still to be rational. For example, with technological advances, unexpected robot behaviours may become less common¹¹—particularly if robots are limited in the scope of their functionality. At the same time, the risks of poor outcomes may be compensated for by the benefits that the enhanced capabilities of robots can bring e.g. being more physically robust, they may able to immobilise rather than kill insurgents (Lokhorst and van den Hoven 2014), and be able to take more measured decisions in the absence of emotional responses. Consequently, in practice civilians may have limited evidence of reasons to distrust a robot force but good reasons to trust them on other grounds e.g. if their lives are at risk.¹² Arguably this could make it sufficiently rational for a civilian to trust a robot force—or at least not to distrust it.

Whether this is actually rational according to Simpson’s account depends on what is conceived to be valid evidence for justified belief e.g. whether evidence needs to reflect an external physical truth or not (e.g. Kelly 2016). In practice the technical sophistication and opacity of such systems—particularly in the context of military technology—means that humans are increasingly unlikely to be aware of and/or understand the real risks of them. As a consequence, they might believe a robot to be capable of being trusted if there was no evidence of perverse outcomes, even if the impact should one happen is extremely high. That is, it may be theoretically rational to trust robots based on the evidence that civilians are likely to have, when in reality, it may not be advisable to do so.

For example, in the context of his account, Cogley (2012) claims that it may be sufficient for a party to merely believe that they are party to a normative relationship with a trustee as the basis for trust—whether or not they actually are. That is, trust may not be dependent on attributes of a trustee that have an objective ontological status as Simpson appears to suggest, but are instead epistemically subjective (Kirkpatrick et al. 2017).¹³ Such expectations might be unrealistic, but could be the basis of trust nonetheless. In similar vein, Coeckelburgh (2009) claims that trust may not be based on the trustor’s assessment of the plausibility of the trustee but simply on perceptions of emotional bonds between them. Trust on these grounds would therefore be an emergent property of social relationships and whether a robot force could be trusted would therefore depend on whether they were perceived to be part of those interactions—which would be possible simply on the basis of them appearing to have the characteristics of a social entity regardless of whether they actually had them. He argues that this is consistent with the way that we treat other humans—as in practice we do not have evidence that even they have the capacities to reason, have motives, or be autonomous because all we see is their outward behaviours—but we trust them nonetheless. Whether such trust may be considered rational depends again on exactly what is conceived of as valid evidence for justified belief—though again, this debate is beyond the scope of this paper.

There is certainly empirical evidence that in practice humans appear to respond to, and potentially trust robots despite a lack of evidence that they actually have the necessary capabilities to be plausible trustees—and therefore whether or not this is ‘foolish’ as Simpson claims. Whether these responses are based on affective responses, irrational or rational beliefs, irrational or rational responses to evidence (or lack of it), other causes, or a combination of factors is not always distinguished however.

For example, Nass et al. (2000) found that that when we encounter an entity that behaves like a human our brains default to treating it as such, ‘mindlessly’ applying social rules and social behaviours such as politeness and reciprocity without even being aware of doing so, and even responding to indicators of ‘personality’. Surprising as this seems, these phenomena have been observed even when participants deny that the technology that they were dealing with had any human traits or characteristics. It is possible that there are alternative explanations of these observations e.g. that participants in their study were thinking of the programmer rather than the computer when responding—though Nass and Moon deny this.

There is also evidence that Artificial intelligence (AI) does not even need to closely mimic humans—either physically or in terms of behaviour—to elicit such responses. For example, Sung et al. (2007) reported that the autonomous Roomba vacuum cleaner, which has neither human-like appearance or behaviours, can trigger social emotions, and Breazeal (2003, p. 151) noted that when a robot can physically mimic even very basic human expressions, humans interacting with it see its actions as “the product of intents, beliefs, desires, and feelings” and respond accordingly.

Such responses may even lead to trust. For example, the ELIZA program was an early form of chatbot that was built to mimic a Rogerian psychotherapist. It did no more than decompose phrases in the input and present them back to the user in a way that would sustain conversation. However, despite this limited functionality, users believed that the program had real understanding of what they were saying—and the creator of the program even found his secretary confiding sensitive personal information to it (Weizenbaum 1984).

In conclusion, the grounds for trust that I have described in this section would allow a trustee to be trusted without necessarily having the capacities that would give them moral status equivalent to that of a human. If trustees can be trusted without any evidence of their attributes i.e. on a purely subjective basis, then this is clearly the case. In the case of trust that is based on some evidence of an actual relationship between trustor and trustee this requires more explanation. That is, in this case a trustee would need to understand the requirements of their relationship with the trustor and respond accordingly i.e. they need to have some capacity for loyalty and hence of differentially valuing actions to at least some extent– but do not require a full capacity for moral reasoning. If moral status is scalar (Jaworska and Tannenbaum 2018) then as Arneson (1999) has suggested, it is not just the capacity to value that makes a difference to moral status, but also the level of this capacity. As such, a trustee with a more limited ability to prioritise actions within the scope of their relationship with a trustor would be accorded less moral status than a trustee who was of good character more broadly and could therefore value a greater range of things, and more fully.

As a consequence, contrary to Simpson’s first claim, it is plausible that alternative accounts of rational normative trust based at least partially on subjective grounds are likely to be more common in the context of a counter-insurgency engagement and could allow robot forces to be trusted whilst still offering a moral saving over the use of human ones. Having established this as a possibility, I next make my case against his second main claim i.e. that such forces would nonetheless be ineffective because their use would undermine civilian trust in those that deployed them.

Nonetheless, evaluating the potential for different forms of trust, and under different conditions, is still a useful exercise as the basis for scenario planning. That is, it allows the implications of these accounts to be considered in advance in case the conditions for them come to be met in the future e.g. through advances in robot capabilities, or engagement in counter-insurgencies amongst civilians with more robo-friendly beliefs. Consideration of such scenarios—even if speculative—allows the potential ethical consequences of such trust to be considered, as well as the structures and measures that might be required to address them. It also provides an opportunity to re-direct the development of the technology if there are unacceptable risks involved.