Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021)

The limitations of scalar rewards and the advantages of vector-based multi-objective rewards for computational agents have been extensively established in prior work [41, 60, 62]. In the interests of space and brevity, we focus here on the aspects that are of most relevance to the reward-is-enough hypothesis.

Clearly, many of the tasks faced by an intelligent decision-maker require trade-offs to be made between multiple conflicting objectives. For example a biological agent must aim to satisfy multiple drives such as reproduction, hunger, thirst, avoidance of pain, following social norms, and so on. A computational agent does not have the same physical or emotional motivations and so, when applied in the context of a highly-constrained task such as playing a board game like Go, there may be a single, clearly defined objective. However, it is likely that the agent will need to account for multiple factors in its decision making, when applied in less restricted environments. The ubiquity of multiple objectives is evident even in the cases presented by Silver et al. For example, in Sect. 3 they suggest that a squirrel’s reward may be to maximise survival time, or reproductive success, or to minimise pain, while a kitchen robot may maximise healthy eating by its user, or their positive feedback, or some measure of the user’s endorphin levels. An agent based on scalar rewards must either be maximising only one of these competing objectives, or some scalarised combination of them.

The prevalence of genuinely multi-objective tasks in the real-world is reflected in the thriving nature of research fields such as multi-criteria decision-making [18, 75, 80] and multi-objective optimisation [20, 27]. Furthermore, this is reflected in the broad range of application areas that involve multi-objective aspects, as identified in [19, 41], which span almost all aspects of human society. In particular, any decision which affects multiple stake-holders will require consideration of multiple objectives [21].

Silver et al. acknowledge that multiple objectives can exist, but state that “a scalar reward signal can represent weighted combinations of objectives". In the context of multiple objectives, the agent is concerned with maximising some measure of utility which captures the desired trade-offs between those objectives. While it is certainly true that rewards representing multiple objectives can be combined into a scalar value via a linear weighting, it is well known that this places limitations on the solutions which can be found [24, 79].¹ Furthermore, in many real world problems different objectives operate at significantly different time scales, making it practically impossible to find a meaningful trade-off among them in a single time step of learning (examples from biological intelligence are given in Sect. 4.2). This means that a scalar representation of reward may not be adequate to enable an agent to maximise its true utility [62]. In particular, some forms of utility such as lexicographic ordering cannot be represented as a scalar value [28]. In contrast, intelligence based on vector rewards and approaches that are explicitly multi-objective can directly optimise any desired measure of utility [41].

A second advantage of multi-objective reward representations is that they allow for a greater degree of flexibility in adapting to changes in goals or utility. A scalar reward representing a weighted combination of objectives directly encodes a single, fixed weighting of those objectives, and this restricts the agent to learning only about that weighting. In contrast, an agent using a multi-objective representation can follow behaviour which is optimal with respect to its current goal, while simultaneously performing learning with regard to other possible future goals or utility preferences. This allows for rapid or even immediate adaptation should the agent’s goals or utility change, which we would argue is likely to arise in dynamic environments, particularly in the context of life-long learning. This approach, which is known in multi-objective reinforcement learning research as multi-policy learning [62], cannot readily be achieved by an agent observing only a scalar reward. Silver et al. themselves state that “Intelligence may be understood as a flexible ability to achieve goals", but scalar rewards are not sufficient to provide the degree of flexibility supported by multi-policy multi-objective methods.

Finally we wish to address Silver et al.’s comment that “a solution to a specialised problem does not usually generalise; in contrast a solution to the general problem will also provide a solution for any special cases". We disagree with the implied assumption that maximisation of a cumulative scalar reward is the general case. Scalar rewards (where the number of rewards \(n=1\)) are a subset of vector rewards (where the number of rewards \(n\ge 1\)). Therefore, intelligence developed to operate in the context of multiple rewards is also applicable to situations with a single scalar reward, as it can simply treat the scalar reward as a one-dimensional vector. The inverse is not true – while a vector reward can be mapped onto a scalar reward, this inevitably involves a loss of information, which will limit some capabilities of the intelligence, as discussed in the previous two paragraphs. Therefore it is clear that problems with scalar rewards are in fact the special case. A similar argument can be made regarding the generality of risk-aware decision-making compared to single-objective decision-making.

Section 3 of Silver et al. identifies the following set of intelligent abilities which they assert could arise by maximising a scalar reward: Knowledge and learning, Perception, Social intelligence, Language, Generalisation, Imitation, and General intelligence.

For some abilities such as knowledge and learning, and imitation, we concur with the reward-is-enough hypothesis. While multi-objective rewards may provide benefits relating to these areas such as improving efficiency, they are not strictly required in order for an intelligent agent to develop these abilities. However, we contend that other abilities are clearly multi-objective in nature. In the following subsections we will address the benefits which multi-objective rewards may provide over scalar rewards for each of these aspects of intelligence. We believe that the issue of general intelligence merits a deeper discussion, particularly in regard to the creation of artificial general intelligence, and so we will defer discussion of this facet of intelligence until Sect. 6.

When considering natural intelligences it is important to draw a clear distinction between the evolutionary process which has produced these intelligences, and the functioning of the intelligences themselves. Biological evolution optimises for a single objective (i.e. reproductive fitness), to an environment that varies over time and space (though its operationalisation depends greatly on the environment [42, 70]). In contrast, at at least five distinct levels of analysis for human organisms, an organism’s computational processes include multiple objectives that must be balanced during action selection (see Fig. 1).

In optimising for reproductive fitness, evolution generates an organism’s genotype (Fig. 1), which in turn creates what we could call an intellectual phenotype, a set of innate intellectual capacities that an organism can use to learn about and interact with its environment. In mammals, as in most other organisms, the bio-computational processes that constitute that intellectual phenotype have no single objective; rather, they include multiple objectives including hunger satiation [29], thirst quenching, social bonding [59], and sexual contact [34], as described in Fig. 1.

Even if reproduction is regarded as a ‘single objective’ of the evolutionary process, broadly construed, at the organism level it is not a primary reinforcer at all. Rather, the organism’s phenotype includes a set of features, including intelligent systems, which have been tuned by the evolutionary process because they tend to lead to environmental fitness and ultimately genetic reproduction. This includes sexual orgasm [34], social contact with conspecifics like offspring [59], and so on. These do not necessarily internally store any kind of explicit representation of reproduction as a goal. Rather reproduction is an emergent result of the organism fulfilling other multiple primary objectives. Understanding the multi-objective nature of motivation in biological systems is critical because different systems are more dominant at different times, depending on context. An example in this regard concerns the remediation of pain and hunger. On the one hand, pain warrants an immediate response, while on the other hand, for hunger this response can be delayed for hours up to days. These examples demonstrate that such objectives could not be aggregated in a single reward signal, due to the difference in time scale in which these rewards are relevant, and we note that distinct biological subsystems are in place to support these differences. Simply understanding the computational processes of the organism is insufficient to predict behaviour without also including an account of the state of the organism and its environment, and the relevant objectives that arise as a function of that state.

The clearest example of biological primary objectives might be what are called ‘primary reinforcers’ in behavioural psychology. These constitute behavioural goals that are innate [29], such as hunger satiation, thirst quenching, and sexual satisfaction (Fig. 1, ‘behavioural level’). These are themselves entire families of objectives, because an organism needs a wide array of nutrients to survive and can pursue very specific objectives to ensure that each nutrient is obtained. Even for specific primary reinforcers, there may be multiple biological signals acting as proxy objectives to ensure those objectives are obtained. For instance, salt consumption alone activates taste receptors, interoceptive signals related to ingestion, and blood osmolality detectors designed to maintain homeostatic balance [72]. These receptors drive the value assigned to consuming salt during decision-making at any particular moment. Thus, even for just a single critical nutrient, there are multiple biological objectives [55, 82] on multiple level tuned to ensure an appropriate level of consumption.

At a psychological level (Fig. 1), humans appear to hold multiple irreducible and irreconcilable moral [38] and life [68] objectives. Moral objectives include preferences for equality, for harm avoidance, and upholding authority hierarchies and group loyalties [38] and may have a basis in distinct biological tendencies. Schwartz et al. [68] identified ten distinct human values, including benevolence, security, hedonism, and autonomy that appear to be distinct life objectives for people across cultures to varying degrees.

Proponents of a single-objective account of RL in biological organisms might look for a single neurotransmitter release that could conceivably underlie all the objectives outlined above – perhaps a global reward signal or brain region that combines all the objectives outlined in the previous section into a single process. However, even if such a mechanism was identified as, for instance, release of dopamine to indicate reward prediction error, the reward-is-enough hypothesis would not follow: that single signal is properly thought of as hard-coded to achieve multiple objectives, as outlined in Fig. 1.

Dopamine is often associated with reward, but it is more appropriate to characterize it as a ‘reward prediction error’ (RPE) signal than as a simple signal of positive reinforcement [54] (Fig. 1, ‘Neurochemical level’). The reward system releases dopamine as a response to a signal indicating a reward is coming [67], but on release of a reward, dopamine is only delivered if the reward was unexpected. It might be said only a single neurotransmitter performs the RPE function in mammalian reward learning, but this is not the same as saying mammalian RL is single-objective, because the RPE signal is delivered to drive learning across a wide variety of domains. For instance, not only is dopamine responsible for modulating goal-directed behavior, but it also seems to play a role in learning within sensory systems including visual, auditory, olfactory, and taste cortices [53]. Additionally, other neurochemicals like oxytocin [52] or serotonin [81] are important in the experience of pleasure, attachment, and motivation, and although these may ultimately depend on the dopaminergic system for motivational power, their presence demonstrates the neurochemical hardcoding of a variety of objectives in animal behavior [52, 67, 81].

A number of neuroanatomical brain regions are important for the production and assessment of value calculation and reward (Fig. 1, ‘Computational level’). A full survey of biological decision-making is outside of the scope of this article. But in brief, subcortical regions like the nucleus accumbens are thought to be involved in reward processing [45], while the ventromedial prefrontal cortex (vmPFC) appears to formulate value signals associated with potential rewards. However, these do not appear to function as parts of single-objective reinforcement learning systems because the values represented are context dependent [65]. When an organism is hungry, the value of food, as recorded in the vmPFC, is higher [74]; when an organism is thirsty, the value of drinking is higher. Something like a ‘common currency’ might indeed be represented in the vmPFC [51] and related regions, but to extend the metaphor, the ‘exchange’ rate between that currency and various physiological and psychological goals and drives changes currently based on context, limiting the applicability of any single-objective account.

Byrnes [15] attempts a description of the brain as an integrated reward learning system. The model described draws on Mollick et al.’s [54] description of the brain’s phasic dopamine system as well as classic work describing the brain as an array of parallel loops [5]. Here, three separate categories of phasic dopamine signals are described: reward prediction error for basic universal goals, reward prediction error for motor action execution, and supervised learning error signals. These are distinct processes required for human intelligence, all with their own objectives, but they are all required for a human brain (or a mammalian brain more generally) to function correctly.

One of the main arguments presented by Silver et al. is that the maximisation of even a simple reward in the context of a suitably complex environment (such as those which occur in the natural world) may suffice for the emergence of general intelligence. They illustrate this via the following scenario:

For example, consider a signal that provides +1 reward to the agent each time a round-shaped pebble is collected. In order to maximise this reward signal effectively, an agent may need to classify pebbles, to manipulate pebbles, to navigate to pebble beaches, to store pebbles, to understand waves and tides and their effect on pebble distribution, to persuade people to help collect pebbles, to use tools and vehicles to collect greater quantities, to quarry and shape new pebbles, to discover and build new technologies for collecting pebbles, or to build a corporation that collects pebbles. [70, p.10]

While the development of open-ended, far-reaching intelligence from such a simple reward is presented positively by Silver et al., this scenario is strikingly similar to the infamous paperclip maximiser thought experiment from the AI safety literature [10]. In this example, a superintelligent agent charged with maximising the production of paperclips enslaves humanity and consumes all available resources in order to achieve this aim. While unrestricted maximisation of a single reward may indeed result in the development of complex, intelligent behaviour, it is also inherently dangerous [56]. For this reason, AI safety researchers have argued in favour of approaches based on satisficing rather than unbounded maximisation [73], or on multi-objective measures of utility which account for factors such as safety or ethics [77].

Even when safety is not at risk, it is vital to carefully consider which types of behaviour can arise when deploying learning agents in society. Depending on the application, elements such as bluffing or information hiding can potentially be harmful and undesirable. For example, consider a smart grid setting, in which autonomous agents decide on behalf of the homeowners how to best store or trade locally generated green energy. This is, in general, a competitive scenario. While the main goal of each agent is to ensure the comfort and reduce the costs of the household, it is not acceptable for agents to manipulate the market or submit bluff-bids, even if these behaviours would be optimal with respect to their reward signal. Since autonomous agents need to operate in the context of our society, we should give careful thought to and investigate what type of emergent behaviour is desirable. This will be important to avoid the development of selfish and harmful mechanisms, under the pretext of being optimal with respect to a single numerical feedback signal.

Therefore we argue that even if scalar rewards are enough for the development of general intelligence, they are not sufficient for the far more important task of creating human-aligned AGI. While safety and ethics are not the focus of Silver et al.’s paper, it is concerning that these issues are not acknowledged in a paper which is actively calling for the development of AGI. More broadly we note that Silver et al. do not provide guidance as to the actual objective or reward signal which may be suitable for the creation of human-aligned AGI.

Even if we were to accept the hypothesis that an intelligent agent’s behaviour can arise from maximising a single scalar reward within the environment, that does not necessarily imply that the design and construction of such a scalar reward is feasible. In this sense, the pebble-collecting scenario is an outlier as the agent has access to a straightforward, dense source of reward (“+1 reward... each time a round-shaped pebble is collected"). In practice the sort of problems for which general intelligence is required will rarely lead to such simple and immediate feedback, and the design of a suitable reward will be a major challenge. In addition analysing the behaviour of a general intelligence in terms of a single reward is unlikely to provide sufficient insight into the drivers of that behaviour. For example, interpreting the behaviour of a squirrel as maximising a single scalar reward representing “survival" does not help to understand the mechanisms of that reward, nor does it assist in the construction of an equivalent reward signal that induces squirrel-like behaviour in an arbitrary intelligent agent.

As described by Amodei et al. [7], reward specification is difficult even in trivial systems, and reward misspecification and reward hacking often lead to surprising, unintended, and undesirable behaviour. In more complex systems with more general agents, the potential for reward misspecification is significantly increased [31]. We argue, then, that the application of a single scalar reward signal leads to significant risk of unpredictable and undesirable behaviour. Given the limitations of their human designers, scalar rewards will most likely not be enough for the development of AGI with guaranteed behavioural properties, and predictable reward design is better achieved using multi-objective methods.

In order for general artificial intelligence to be beneficial to humanity, it will need to be accountable and adaptable to human ethics, as well as human needs and aims. To be a part of society, an agent needs to adapt, and be accountable to others in this society, and we argue that agents that optimise for a single objective are severely handicapped in doing so. In this section, we will first explain the need for multiple objectives from an ethical and human-alignment point of view, and second explain our vision for future AI systems that form an integral part of society; agents that can be reviewed with respect to, and adapted to, a changing society.

Ultimately, all ethical capabilities that AGI can attain will have to come from humans, as well as all other goals and aims. There simply is no other available source for ethics and general goals than humans. However we do not agree amongst ourselves what ethically optimal behaviour is. Philosophers – arguably the closest there is to generalist experts in ethics – disagree on a wide range of ethical questions including meta-ethics, moral motivation, and normative ethics [12], and their positions on these disagreements correlate systematically with personal identities including race and gender. Psychologically, moral intuitions seem to arise from a pluralistic set of incommensurable and innate moral values that differ systematically across different political parties [36]. Fundamental human psychological values differ individually and across cultures [68]. Furthermore, even if it is not a question of ethics, the question of what to care about is not trivial either [35]. Therefore, we cannot expect people who need to specify the rewards for an AI system to get it right, especially not in one go. Moreover, priorities are likely to change over time. Any AI system – general or not – that is deployed for a long time, and that is possibly propagated to new generations of the systems, must be able to deal with this.

Humans who will have to specify what the agent must care about will likely identify multiple goals that are important. For example, a self-driving car will need to minimise travel time, minimise energy consumption, as well as minimise the expected harm resulting from a trip to both the occupants of the car, other humans in the environment, animals, and property. Furthermore, a reasonable trade-off between these objectives (which may well be a non-linear one, as we explained in Sect. 3.1) must be identified during training. However, the engineers of such a system are most likely not ethicists nor legal experts. Furthermore, they do not own the utility of the system’s deployment, and may not ultimately bear the responsibility for the system being deployed in practice. Therefore, the human-alignment process will necessarily be in the hands of others, who need a clear explanation from the AI system about the available trade-offs between objectives in different situations. We therefore believe that an explicitly multi-objective approach is necessary for responsible agent design. For further arguments why human-aligned AI is inherently multi-objective, please refer to [77].

Once the design, training, and testing is completed, and the AI system is deployed, it will be equipped with a set of objectives, and a mechanism (e.g., a utility function [63]) for making online decisions about trade-offs between these objectives. However, it may well encounter situations that were not foreseen in training, and the trade-offs between objectives in these situations become unlike hitherto encountered trade-offs. Here we encounter another key benefit of an explicitly multi-objective approach – and in our opinion an ethically necessary capability, of any agent. In combination with probabilistic methods, the agent can identify when it becomes too uncertain about which trade-offs will be desired and therefore, if possible, defer the decision back to responsible humans, or shut down safely. When this is not possible, these situations can be identified and reported back. For example, if a parrot suddenly gets loose in an automated factory, and the AI system correctly identifies it as an animal, but has never had to make choices between animal safety, human safety, product damage, and production before, the system may try to opt for safe system shutdown if a responsible human cannot be reached. However, if the parrot flies straight into the production line, more drastic immediate action might be required, and there is no time to shut down safely. These immediate actions will necessarily be taken on the fly, and will have to be reported and later reviewed to see whether the taken actions were indeed what the responsible humans would have wanted the system to do.

Finally, when undesirable things have occurred, an agent needs to be able to explain the decision made. Single objective systems are only able to provide simple details such as what was the perceived state and that its chosen action maximised its reward [22]. Such explanations provide little understanding to the users. However, an explicitly multi-objective approach confers significant further benefits. Namely, it can help diagnose exactly what went wrong, such as: what was the trade-off between objectives; what alternative outcomes would have occurred with alternative trade-offs; or, was the selected policy providing an undesired trade-off between objectives [26]. Hence, a multi-objective approach allows explicitly attributed details, as well as contrastive and counterfactual explanations of the reasoning process behind behaviour rather than just the outcome [25]. Being able to explicitly identify the trade-off underlying an agent’s basis for reasoning has long been argued as a key component of transparency in AGI [43].

One possible implementation of these concepts would be a review-and-adjust cycle [41]. Figure 2 demonstrates how a review-and-adjust cycle could be applied when developing an agent for a multi-objective Markov decision process (MOMDP). During the planning or learning phase, an AGI would utilise a multi-objective algorithm to compute a set of optimal policies for all possible utility functions. In the selection phase, a policy is selected to be executed, possibly with direct or indirect user feedback. The selected policy is then executed, during the execution phase. The outcome from the policy execution can then be reviewed by the overseer (either a human, the AGI itself, or another AGI), along with the AGI’s explanation of its policy selection. The MOMDP, utility function or set of solutions can then be updated based on this review. We note that such reviews can not only be triggered by incidents, but also by regular inspection.

We see such a review-and-adjust cycle as an essential feature of future AI. We, as AI researchers have to enable responsible deployment, and well-informed review of the systems we create is a key feature of this. It is our opinion that the above-mentioned benefits are not merely desirable, but that it is a moral imperative for AI designers to obtain them, in order to create AI systems that more likely benefit society. In addition to the mathematical, technical, and biological arguments for why scalar reward is not enough, we thus also point out that there are ethical and societal reasons why scalar rewards are not enough.

We acknowledge the difficulties which may arise in implementing human oversight of AGI if the latter achieves superhuman levels of intelligence [6, 11, 33]. Superintelligent AGI may be motivated to, and highly capable of, deceiving human overseers, or its behaviours and reasoning may simply be too complex for human understanding. Nevertheless we would argue that it is certainly preferable to attempt such oversight than not, and that a multi-objective AGI will provide greater transparency than a single-objective AGI. Nevertheless, we acknowledge that such an approach will not necessarily guarantee a manageable superhuman AGI, and therefore a careful ethical consideration of such research efforts is warranted.