Foundations of Trusted Autonomy

Foundational theories have contributed greatly to scientific progress in many fields. Examples include Zermelo-Fraenkel set theory in mathematics, and universal Turing machines in computer science. Universal Artificial Intelligence (UAI) is an increasingly well-studied foundational theory for artificial intelligence, based on ancient principles in the philosophy of science and modern developments in information and probability theory. Importantly, it refrains from making unrealistic Markov, ergodicity, or stationarity assumptions on the environment. UAI provides a theoretically optimal agent AIXI and principled ideas for constructing practical autonomous agents. The theory also makes it possible to establish formal results on the motivations of AI systems. Such results may greatly enhance the trustability of autonomous agents, and guide design choices towards more robust agent architectures and incentive schemes. Finally, UAI offers a deeper appreciation of fundamental problems such as the induction problem and the exploration-exploitation dilemma.

This chapter discusses the topic of Goal Reasoning and its relation to Trusted Autonomy. Goal Reasoning studies how autonomous agents can extend their reasoning capabilities beyond their plans and actions, to consider their goals. Such capability allows a Goal Reasoning system to more intelligently react to unexpected events or changes in the environment. We present two models of Goal Reasoning: Goal-Driven Autonomy (GDA) and goal refinement. We then discuss several research topics related to each, and how they relate to the topic of Trusted Autonomy. Finally, we discuss several directions of ongoing work that are particularly interesting in the context of the chapter: using a model of inverse trust as a basis for adaptive autonomy, and studying how Goal Reasoning agents may choose to rebel (i.e., act contrary to a given command).

In this chapter, we describe social planning mechanisms for constructing and representing explainable plans in human-agent interactions, addressing one aspect of what it will take to meet the requirements of a trusted autonomous system. Social planning is automated planning in which the planning agent maintains and reasons with an explicit model of the other agents, human or artificial, with which it interacts, including the humans’ goals, intentions, and beliefs, as well as their potential behaviours. The chapter includes a brief overview of the challenge of planning in human-agent teams, and an introduction to a recent body of technical work in multi-agent epistemic planning. The benefits of planning in the presence of nested belief reasoning and first-person multi-agent planning are illustrated in two scenarios, hence indicating how social planning could be used for planning human-agent interaction explicitly as part of an agent’s deliberation.

A multi-agent architecture called the Adaptive Team of Agents (ATA) is introduced, wherein homogeneous agents adopt specific roles in a team dynamically in order to address all the sub-tasks necessary to meet the team’s goals. Artificial neural networks are then trained by neuroevolution to produce an example of such a team, trained to solve the problem posed by a simple strategy game. The evolutionary algorithm is found to induce the necessary in situ adaptivity of behavior into the agents, even when controlled by stateless feed-forward networks.

In an increasingly complex and interconnected world, there is an increasing need for autonomous systems that can control systems that are beyond the capabilities of human operators. One of the key issues to be addressed in developing trusted autonomous systems is dealing with the phenomenon of “emergence”; either by taking advantage of emergence or avoiding emergence. Swarm intelligence systems, based on the interaction of a large number of relatively simples agents, rely on emergent intelligence for their problem solving capabilities. When used in trusted autonomous systems, the emergent behaviour of swarm intelligence systems can be both a blessing and a curse.

Game play by humans has always required autonomy and trust. Autonomy because a person chooses to play and takes in-game actions; and trust that the other players adopt a lusory attitude and abide by the rules of the game. The chapter highlights several areas of digital (computer) game design and development that will be revolutionised by the technology of, and framework of Trusted Autonomy: TA game AI that display emotional and other (non-logic) forms of intelligence; TA games that are aware of themselves and of the player, and self-modify to enhance play; TA communities that create a safe, fulfilling, and non-toxic environment; and TA augmented reality games that keep the player and wider community safe during play.

As robots become increasingly common in a wide variety of domains—from military and scientific applications to entertainment and home use—there is an increasing need to define and assess the trust humans have when interacting with robots. In human interaction with robots and automation, previous work has discovered that humans often have a tendency to either overuse automation, especially in cases of high workload, or underuse automation, both of which can make negative outcomes more likely. Frthermore, this is not limited to naive users, but experienced ones as well. Robotics brings a new dimension to previous work in trust in automation, as they are envisioned by many to work as teammates with their operators in increasingly complex tasks. In this chapter, our goal is to highlight previous work in trust in automation and human-robot interaction and draw conclusions and recommendations based on the existing literature. We believe that, while significant progress has been made in recent years, especially in quantifying and modeling trust, there are still several places where more investigation is needed.

Effective robots and autonomous systems must be trustworthy. This chapter examines models of trustworthiness from a philosophical and empirical perspective to inform the design and adoption of autonomous systems. Trustworthiness is a property of trusted agents or organisations that engenders trust in other agent or organisations. Trust is a complex phenomena defined differently depending on the discipline. This chapter aims to bring different approaches under a single framework for investigation with three sorts of questions: Who or what is trustworthy?–metaphysics. How do we know who or what is trustworthy?–epistemology. What factors influence what or who should we trust?–normativity. A two-component model of trust is used that incorporates competence (skills, reliability and experience) and integrity (motives, honesty and character). It is supposed that human levels of competence yield the highest trust whereas trust is reduced at sub-human and super-human levels. The threshold for trustworthiness of an agent or organisation in a particular context is a function of their relationship with the truster and potential impacts of decisions. Building trustworthy autonomous systems requires obeying the norms of logic, rationality and ethics under pragmatic constraints–even though there is disagreement on these principles by experts. Autonomous systems may need sophisticated social identities including empathy and reputational concerns to build human-like trust relationships. Ultimately transdisciplinary research drawing on metaphysical, epistemological and normative human and machine theories of trust are needed to design trustworthy autonomous systems for adoption.

The relationship between trust and uncertainty has not been fully developed in current frameworks on trust, including trust in autonomous systems. This chapter presents an investigation of this relationship. It begins with a survey of trust and distrust in general, followed by a focus on human-robot interaction (HRI). Thereafter, the roles of uncertainty in trust and distrust are elucidated, and the impacts of different kinds and sources of uncertainty are elaborated.

Information systems in the early 21st Century have become a critical enabler of increased value to the business, or as people in Defence might call a ‘force multiplier’.

We investigated if an autonomous system can be provided with reasoning that maintains trust between human and system even when human and autonomous system reach discrepant conclusions. Tversky and Kahneman’s research [27] and the vast literature following it distinguishes two modes of human decision making: System 1, which is fast, emotional, and automatic, and System 2 which is slower, more deliberative, and more rational. Autonomous systems are thus far endowed with System 2. So when interacting with such a system, humans may follow System 1 unawares that their autonomous partner follows System 2. This can easily confuse the user when a discrepant decision is reached, eroding their trust in the autonomous system. Hence we investigated if trust in the message could interfere with trust its source, namely the autonomous system. For this we presented participants with images that might or might not be genuine, and found that they often distrusted the image (e.g., as photoshopped) when they distrusted its content. We present a quantum cognitive model that explains this interference. We speculate that enriching an autonomous system with this model will allow it to predict when its decisions may confuse the user, take pro-active steps to prevent this, and with it reinforce and maintain trust in the system.

Errors are the enemy of classification systems, so minimising the total probability of error is an understandable objective in statistical machine learning classifiers. However, for open-world application in trusted autonomous systems, not all errors are equal in terms of their consequences. So, the ability for users and designers to define an objective function that distributes errors according to preference criteria might elevate trust. Previous approaches in cost-sensitive classification have focussed on dealing with distribution imbalances by cost weighting the probability of classification. A novel alternative is proposed that learns a ‘confusion objective’ and is suitable for integration with modular Deep Network architectures. The approach demonstrates an ability to control the error distribution in training of supervised networks via back-propagation for the penalty of an increase in total errors. Theory is developed for the new confusion objective function and compared with cross-entropy and squared loss objectives. The capacity for error shaping is demonstrated via a range of empirical experiments using a shallow and deep network. The classification of handwritten digits from up to three independent databases demonstrates desired error performance is maintained across unforeseen data distributions. Some significant and unique forms of error control are demonstrated and their limitations investigated.

The Collaborative Advanced Robotics and Intelligent Systems (CARIS) laboratory at the University of British Columbia studies the development of robotic systems that are capable of autonomous human-robot interaction. This chapter will provide an overview of our laboratory’s activities and methodologies. We first discuss a recently-concluded multi-institutional three year project to develop autonomous robot assistants which aid in assembly operations at manufacturing facilities. Next we discuss the primary methodology employed by our laboratory, by which we identify communicative cues used in interactions between people, describe these cues in detail, then implement and test them on robots. This is followed by an overview of recent communicative cue studies carried out by our group and our collaborators. We conclude by discussing current and future work.

In this chapter, I argue that agents need human-like intrinsic motivation in order to achieve true autonomy, which is especially important for dealing with complex, uncertain, or unpredictable environments. A computational cognitive architecture is presented that incorporates not only cognitive capabilities necessary for the functioning of agents but also intrinsic (as well as derived) motivation for agents. With this model, an agent is able to function properly and autonomously in complex environments, as demonstrated by a wide range of simulations.

Computational motivation—such as curiosity, novelty-seeking, achievement, affiliation and power motivation-facilitates open-ended goal generation by artificial agents and robots. This further supports diversity, adaptation and cumulative, life-long learning by machines. However, as machines acquire greater autonomy, this may begin to affect human perception of their trustworthiness. Can machines be self-motivated, autonomous and trustworthy? This chapter examines the impact of self-motivated autonomy on trustworthiness in the context of intrinsically motivated agent swarms.

Given what has been discovered in the case of human cognition, this principle seems plausible: An artificial agent that is both autonomous (A) and creative (C) will tend to be, from the viewpoint of a rational, fully informed agent, (U) untrustworthy. After briefly explaining the intuitive, internal structure of this disturbing (in the context of the human sphere) principle, we provide a more formal rendition of the principle designed to apply to the realm of intelligent artificial agents. The more-formal version makes use of some basic structures available in one of our cognitive-event calculi, and can be expressed as a (confessedly — for reasons explained — naïve) theorem. We prove the theorem, and provide simple demonstrations of it in action, using a novel theorem prover (ShadowProver). We end by pointing toward some future defensive engineering measures that should be taken in light of the theorem.

This chapter will use a demonstration scenario, with intersecting vignettes, to investigate a path to where, by 2030, the promise of autonomous systems will have been realised. Like many steps humans have taken in technological evolution, some will be deliberate and cautious whereas others will happen out of necessity - perceived or real. Looking back through historical scenarios can provide an in-sight to the steps taken, their triggering events and the temporal factors, in order to identify potential futures. The crucible of conflict has always been a fertile ground for evolutionary change, particular those pushing the boundaries of moral and ethical thinking. From the flimsy flying machines at the beginning of World War One humanity saw the development of transcontinental aircraft and rockets in a little over three decades, culminating in the first operational use of an atomic weapon. High levels of digitisation and automation became central to the most capable weapon systems where human interaction could often result in degraded system performance, occasionally with disastrous consequences. Ongoing war-fare and disregard for the global rules based order enabled an environment where the combining of cyber and social media capabilities developed into crude but effective ‘trusted autonomous command and control’ system. Like all such steps taken before, this was embraced by global major powers and exploited to its maximum potential.

Being able to trust your teacher has been a pivotal assumption within the training systems. The advent of autonomous systems capable of delivering training in innovative and traditional ways creates a number of questions. The premise of this book allows us to examine how autonomous systems, that is non-human, will impact training and learning environments. The following section seeks to explore the future of trusted autonomy within a training context through both an extrapolation of current trends and creative thought.

This chapter describes the nature of the space environment that makes autonomous space systems a desirable application; describes the various types of space activities in near-Earth and deep space missions, and examples of autonomous systems deployed in space to date; outlines the current state-of-the-art of the intersection between trusted autonomous systems and autonomous space systems; and then presents a variety of possible future trusted autonomous space scenarios.

This chapter considers autonomy as an economic concern, meaning as a matter of determining the outcomes from different ways of allocating scarce resources in a social setting. Specifically, it concerns the outcomes from decision-making mechanisms subject to capital rationing - constraints on available resources - under conditions of fundamental uncertainty. This kind of uncertainty has long been a topic of economic study: epistemic uncertainty distinguishes stochastic risk from uncertainty, while ontological uncertainty equates to non-stationarity and non-regularity in a dynamical systems sense. I also argue that this deeper ontological uncertainty is a manifestation of mathematical incompleteness and unsolvability, which are inherent limitations in reasoning due to logical paradox. Given that non-linear dynamics classifies different degrees of uncertainty in different classes, I propose plasticity as the maximum class that an organism, system, organisation or thing may successfully handle, in the sense of surviving in an environment characterised by that class for better than logarithmic time. Machines that manifest plasticity beyond the most benign classes of uncertainty may then be referred to as autonomous, distinguishing them from automations that rely on strong prediction. Economics also studies mechanisms for handling uncertainty and their system-wide as well as individual outcomes. Paradoxically, uncertainty can easily result from the very measures that were intended to deal with uncertainty, so although a strategy might make completely rational sense from the point of view of a single agent, it can just as easily produce dangerously unstable and unpredictable system-wide outcomes. I focus on a certain class of financial strategies in particular, known as ‘barbell’ and ‘dumbbell’ strategies - which divide investment into hedges against intolerable failure and opportunity bets where failure is tolerable - for their apparent applicability across different classes of uncertainty. At the centre of this picture is a requirement for a theory of self by which an agent can determine failure sensitivity and affordable opportunity in high uncertainty environments under partially observable hard resource limits. The limits of self-knowledge together with the complexity of decision-making under hard capital rationing with the possibility of unexpected budget changes appears to imply that autonomous problem solving must be intrinsically social.