Accountability and Control Over Autonomous Weapon Systems: A Framework for Comprehensive Human Oversight

Accountability and responsibility are key concepts in the academic and societal debate on the ethics and politics of Autonomous Weapon Systems. The Group of Governmental Experts (GGE) on emerging technologies in the area of Lethal Autonomous Weapons Systems (LAWS) of the Convention on Certain Conventional Weapons (CCW) of the United Nations (UN GGE LAWS 2018) lists ‘Human responsibility for decisions on the use of weapons systems must be retained since accountability cannot be transferred to machines. This should be considered across the entire lifecycle of the weapon system.’ as one of the possible guiding principles for LAWS. At the same time, many scholars express concerns that Autonomous Weapon Systems will lead to an “accountability gap” or “accountability vacuum”, that is circumstances in which no human can be held accountable for the decisions, actions and effects of Autonomous Weapon Systems (Matthias 2004; Asaro 2012; Asaro 2016; Crootof 2015; Dickinson 2018; Horowitz and Scharre 2015; Wagner 2014; Sparrow 2016; Roff 2013; Galliott 2015). According to the International Committee of the Red Cross (ICRC 2018), to uphold moral responsibility for decisions in the use of force, human control that is ‘meaningful’, ‘effective’ or ‘appropriate’ is needed and should be maintained. The notion of “Meaningful Human Control” is often mentioned as a requirement in the debate on Autonomous Weapon Systems, but a common definition of what this notion means in practice is lacking (Ekelhof 2019). Scholars have been working on defining and operationalizing Meaningful Human Control over the past years. Horowitz and Scharre (2015) were among the first to try to identify essential components for Meaningful Human Control and according to them Meaningful Human Control addresses accountability, moral responsibility and controllability. Roff and Moyes (2016) also characterized Meaningful Human Control based on three temporal layers; before the start of hostilities (ante-bellum), during the hostilities (in bello) and after the hostilities (post bellum). In their view, Meaningful Human Control links accountability systems and the need for responsible design—when the mechanisms in the first two layers fail, there is a need for accountability. However, both Horowitz and Scharre (2015) and Roff and Moyes (2016) do not sufficiently distinguish between the different forms of responsibility involved, mostly define control in terms of a relation between one human operator (or commander) and one technical device, and do not provide a clear definition of Meaningful Human Control.

The notions of accountability, moral responsibility and controllability are often used as high-level and overarching constructs in the Autonomous Weapon Systems debate, but their definition and relations are not always fully clarified. Nor is it clear how to operationalize these notions into working and verifiable requirements for practical use. In this article, we describe the relations between accountability, moral responsibility, controllability and oversight to show how these notions are distinct, but also connected in order to deepen our understanding. We purposely do not repeat the extensive philosophical discussion on Meaningful Human Control and accountability gaps, but aim to operationalize the concept of Human Oversight in a comprehensive approach and to provide concrete recommendations for an oversight process. We propose a framework for Comprehensive Human Oversight based on an engineering, socio-technical and governance perspective on control. By this, we broaden the view on the control over Autonomous Weapon Systems and take a comprehensive approach that may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems.

Responsibility can be forward-looking to actions to come and backward-looking to actions that have occurred. Accountability is a form of backward-looking responsibility and provides an account of events after they occurred. A person needs to be accountable in order to be responsible and to have control to be able to account for his or her actions and an oversight mechanism is needed in order to hold an actor accountable (Caparini 2004; Fitzsimmons and Sangha 2010; Schedler 1999). Control is described in different ways in different disciplines. In the engineering perspective as mechanism to align goals and output of a technical system which is a mechanical or cybernetic view on control (Åström and Kumar 2014; Pigeau and McCann 2002). In the socio-technical perspective as the ability to induce the behaviour of another person (Koppell 2005) and is also applicable to entities and systems. Control can also be viewed from a governance perspective. Pesch (2015) noted that this perspective on control is lacking robust institutionalized frameworks for engineers.

We view Autonomous Weapon Systems as complex socio-technical systems which is best conceptualized as the result of the interaction between technical, socio-technical (human–machine) and governance components. Therefore, we take in this article the conceptual framework proposed by Van den Berg (2015) to characterize the cyber domain (cf. Fig. 1) and adapt it so as to describe the levels of controllability of Autonomous Weapon Systems. In his framework, Van den Berg (2015) describes the cyber domain constituted of three different layers: (1) the inner layer is the technological layer in which the technology is described, (2) the middle layer is the socio-technical layer in which humans and technology interact in activities and (3) the outer layer is the governance layer in which institutions govern these activities.

In this article, we aim to operationalize the concepts of accountability, control and oversight from a design point-of-view and describe how these notions are both distinct and relate to each other. We realize that a rich and long philosophical discussion on these concepts exists. We focus on those aspects of these concepts that already described in literature, describe the relationships between them and apply them to the military domain. Our main claim is that combining the control mechanisms in the technical, socio-technical and governance layer will lead to a new concept of Comprehensive Human Oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems. By combining these three perspectives, we intend to broaden the current narrow view on Meaningful Human Control, which is often defined as the relationship between the human operator and Autonomous Weapon System.¹ By broadening this view to Comprehensive Human Oversight, we also consider the governance aspects concerning Autonomous Weapon Systems and in the second part of the paper we identify at least two gaps in the control mechanisms of Autonomous Weapon Systems.

The remainder of this article is as follows. Section 2 describes the relationship between responsibility and accountability by characterizing responsibility as being either forward-looking, backward-looking or active (Pesch 2015; Van de Poel 2011). We note that accountability is a component of backward-looking responsibility. Next, we focus on Mark Bovens (2007) notion of “mechanisms of accountability” and thereby identify three possible accountability gaps for Autonomous Weapon Systems based on the three layers identified by Van den Berg (2015). In Sect. 3, we show how accountability and control are linked and provide an engineering, socio-technical and governance perspective on control. We describe how combining these three perspectives constitute the notion of Comprehensive Human Oversight. Section 4 presents the Comprehensive Human Oversight Framework, consisting of three horizontal and three vertical layers (nine blocks) and their connections. In Sect. 5, we apply the Dutch military control instruments that are currently used to the nine blocks of the Comprehensive Human Oversight Framework. Section 6 shows the applicability of the Comprehensive Human Oversight Framework to Autonomous Weapon Systems. This reveals at least two gaps in the control mechanisms in relation to Autonomous Weapon Systems. In Sect. 7 we conclude by identifying three first options for the design of a control mechanism in the technological, the socio-technical and the governance layer as future work in order to achieve Comprehensive Human Oversight and ensure accountability over Autonomous Weapon Systems.

In this section we first define the notion of accountability in relation to the broader concept of moral responsibility. Next, we describe accountability gaps or vacuums and conclude by presenting three possible accountability gaps for Autonomous Weapon Systems.

Responsibility can be forward-looking to actions to come and backward-looking to actions that have occurred. Van de Poel (2011) focusses on moral responsibility for consequences to describe the notions of forward- and backward-looking responsibility and does not describe organizational, social and legal responsibility or responsibility for actions. Two varieties of responsibility that are primarily forward-looking are: (1) responsibility as virtue and (2) the moral obligation that something is the case; and three varieties that are primarily backward-looking are: (3) accountability, (4) blameworthiness and (5) liability.

More formally, forward-looking responsibility is defined by Van de Poel (2011, p. 41).

This statement entails that being responsible includes being accountable or blameworthy. The notion of fitting refers to the appropriateness for someone to hold another accountable under certain conditions. The conditions for which it is appropriate or fitting to hold A backward-looking blameworthy are (Van de Poel 2011):

The definitions of Van de Poel (2011) above describe a form of responsibility that is virtue-based responsibility. A second form is of responsibility is accountability in the way that X is accountable for Y for Z. In this sense accountability performs functions of scrutiny, for example calling someone to account, requiring justifications and imposing sanctions (Mulgan 2000).

These forms of responsibility are conceptually and casually related in many ways. For instance, once can arguably be deemed to be a responsible person (virtue) only if she accepts blame and liability when needed and is willing to account for his or her actions (Williams 2008). A general capacity for accountability is arguably the basis for other forms of backward-looking responsibility, including blameworthiness (Gardner 2007). Moral blameworthiness (in the form of culpability or fault) grounds many forms of criminal and tort liability. And by encouraging accountability, it is probably possible to make persons more able and willing to discharge their moral and social obligations (Pesch 2015). However, these forms of responsibility are also distinct and require different conditions to apply. For instance, Van de Poel (2011) states that an agent can have backward-looking responsibility (i.e. being accountable or blameworthy) without being forward-looking responsible for preventing that state-of-affairs. Also, blameworthiness requires that an agent has unjustifiably and inexcusably committed a wrong action. Whereas accountability simply requires the agent to explain her behaviour, possibly but not necessarily with the goal of showing that it was not wrong, or that thought wrong, given the circumstances, justifiable or excusable (Gardner 2007). Also, Pesch (2015) discussed the concept of “active responsibility” of engineers. Active responsibility could be viewed as forward-looking responsibility as it proactively requires engineers to take societal values of technology into account during the development of technology. It is also paired with ‘passive’ responsibility, also referred to as accountability. The pairing of active responsibility and passive responsibility creates a proactive feedback loop of responsibility that is neither strictly forward-looking as backward-looking responsibility and by this, it takes an intermediate position between these two types of responsibility. This proactive feedback loop enables actors to learn and reflect on their actions.

All forms of responsibility are arguably to be encouraged and promoted in order for Autonomous (Weapon) Systems to be designed, introduced, regulated and used in a morally acceptable way, and many different forms of responsibility gaps have to be avoided to prevent negative ethical and societal effects of this introduction and use (Santoni de Sio & Mecacci, under review). However, whereas the relationship between control and blameworthiness has been widely studied in philosophy (Fischer and Ravizza 1998) and the relation between moral and legal culpability, its gaps, and Meaningful Human Control have been studied in relation to Autonomous (Weapon) Systems, an account of the relationship between accountability, its gaps, control and oversight is still missing. In the next sections we start filling this lacuna.

Accountability is a key concept in political science, public management, international relations, social psychology, constitutional law and business administration literature. In the policy domain, the term accountability has two different uses. On the one hand, it is used to praise or criticize the performance of states, organizations, firms or officials regarding policy or decisions in relation to their ability and willingness to give information and explanations about their actions (‘accountability as a virtue’). Typically, in the political discourse, accountability is used to describe the fairness and equitability of good governance in which authorities are being held accountable by their citizens. In this broad sense, accountability encompasses concepts such as transparency, equity, democracy, efficiency, responsiveness, responsibility and integrity. On the other hand, in a narrow sense, accountability is also used to define the mechanisms for corporate and public governance to hold agents and organisations accountable (‘accountability as a mechanism’) (Bovens et al. 2014). Bovens (2007, p. 450) focuses on the latter sense of accountability and defines it as follows: ‘Accountability is a relationship between an actor and a forum, in which the actor has an obligation to explain and to justify his or her conduct, the forum can pose questions and pass judgement, and the actor may face consequences.’ The relationship between an actor and a forum is a key notion in the concept of accountability. If the explanation is inadequate, sanctions may be imposed on the actor by a forum (Bovens 2007; Greer et al. 2016). Figure 2 provides an overview of the relationship between the accountability elements. Accountability is not only scrutiny after the event has occurred, it has also a preventive and anticipatory use for which norms are (re)produced, internalized and adjusted by means of accountability if necessary.

Similarly, in public administration, mechanisms of accountability are described in terms of an agent having to report on his or her activity to an individual, group or other entity which has the ability to impose costs to the agent (Keohane 2003). In this sense, accountability is an agency theory approach in which the relationship between a principal and an agent is described (Hulstijn and Burgemeestre 2014). This concept of accountability as answerability is most used in public administration, but according to Romzek and Dubnick (1987) accountability can play a greater role than answerability alone. It is also linked to the means that public agencies have to manage internal and external expectations of their stakeholders. To manage these internal and external expectations two factors are critical: ‘(1) whether the ability to define and control expectations is held by some specified entity inside or out-side the agency; and (2) the degree of control that entity is given over defining those agency’s expectations’ (Romzek and Dubnick 1987, p. 228). This notion of accountability is linked to control of expectations of the agency.

Depending on the different relationships between different actors and fora, Bovens (2007) distinguishes five types of (narrow) accountability:

Many scholars point to gaps in accountability relationships that will occur in the deployment of Autonomous Weapon Systems. Asaro (2016) argues that the use of emerging technologies, including Autonomous Weapon Systems, with weak or without norms can lead to limited or easily avoidable responsibility and accountability for states and individuals. Sparrow (2016), building on the work of Matthias (2004) and Roff (2013), states that the use of an Autonomous Weapons Systems might risk an ‘responsibility gap’ and it could be problematic to attribute responsibility for actions taken by Autonomous Weapon Systems to operators. Galliott (2015) also mentions the responsibility gap put forward by Sparrow and argues that shifting to forward-responsibility, instead of only backward-responsibility, and a functional sense of responsibility to include institutional agents and the human role in engineering the system, might be a solution to avoid this gap. Crootof (2015) also discusses the accountability gap and notes that with the use of Autonomous Weapon Systems serious violations of international humanitarian law may be committed resulting in a lack of criminal liability for people, including the deployer, programmer, manufacturer and commander, or the weapon system itself. According to Horowitz and Scharre (2015) the potential of an ‘accountability gap’ is the main motivation to implement the principle of Meaningful Human Control. If an Autonomous Weapon System malfunctions and strikes the wrong target it is possible that no human is responsible for the error of the weapon.

Alston (2010) describes these gaps as an ‘accountability vacuum’ in his UN report to the Human Rights Council on targeted killings. He defines targeted killings as ‘… the intentional, premeditated and deliberate use of lethal force, by States or their agents acting under colour of law, or by an organized armed group in armed conflict, against a specific individual who is not in the physical custody of the perpetrator.’ Alston (2010, p. 26) notes that states failed to disclose: ‘…the procedural and other safeguards in place to ensure that killings are lawful and justified, and the accountability mechanisms that ensure wrongful killings are investigated, prosecuted and punished.’ The reason for this accountability vacuum is that the international community cannot verify the legality of the killing, nor confirm the authenticity of the intelligence used in the targeting process or ensure that the unlawful targeted killing results in impunity. Meloni (2016) argues that the accountability vacuum that Alston described in 2010 has been growing ever since. Cummings (2006) notes that an erosion of accountability could be caused by the use of computer decision making systems, because these systems diminish the user’s moral agency and responsibility due to the perception that the automated system is in charge. This could cause operators to cognitively offload responsibility for a decision to a computer. Which in turn creates a moral buffer, meaning a form of distancing and compartmentalizing of decisions, leading to moral and ethical distance and an erosion of accountability.

Offloading responsibility of decisions by operators to Autonomous Weapon Systems may lead to erosion of accountability. We identify three possible accountability gaps on three different layers:

In the next section we describe the link between accountability and control by following Bovens’ (2007) argument that accountability is a form of control, but not all control forms are accountability mechanisms. We characterize control based on an engineering, sociotechnical and governance perspective based on the layers described by Van den Berg (2015) and briefly highlight where these perspectives fall short. Next, we move to the concept of Meaningful Human Control and argue that social institutional and design dimension at a governance level are needed, because accountability requires strong mechanisms for oversight. We look at an oversight mechanism to connect the technical, socio-technical and governance perspective of control in order to improve accountability for the behaviour of Autonomous Weapon Systems.

Several scholars describe the relationship between accountability and control. According to Bovens (2007) there is a fine line between accountability and control. Koppell (2005, p. 97) states that: ‘If X can induce the behavior of Y, it is said that X controls Y‐and that Y is accountable to X.’ Radin and Romzek (1996) link types of accountability relationships to the degree (high or low) and source (internal or external) of control. Koppell (2005) notes that this seems to mix different types of accountability relationships which is in his sense a weakness of this approach. According to Lupia (in Bovens 2007, p. 453): ‘An agent is accountable to a principal if the principal can exercise control over the agent’. Bovens (2007) contests this by stating that although accountability mechanisms are important to control the behaviour of organizations, control in the Anglo-Saxon sense means ‘having power over’ and can be achieved by ‘very proactive means of directing conduct’. Examples of these proactive means are direct orders, laws, regulations and directives. These means are not accountability mechanisms themselves because they are not procedures in which an actor has to justify and explain his or her conduct to a forum. Bovens (2007) concludes by stating that: ‘Accountability is a form of control, but not all forms of control are accountability mechanisms.’ We have seen above that the concept of “Meaningful Human Control” was introduced in the debate on the ethics of Autonomous Weapon Systems, among other things to preserve human accountability. The question then is if human control can ground effective mechanisms of accountability in relation to the behaviour of agents and institutions who deploy Autonomous Weapon Systems. We will argue, that we need to broaden this view toward Comprehensive Human Oversight mechanisms.

Control has traditionally been defined in different ways, depending on application domains. In this section we describe the perspectives from the engineering, socio-technical and governance point of view based on the layers described by Van den Berg (2015).

Combining our analysis of accountability and perspectives on control, with the three layers described used by Van den Berg (2015) to characterize the cyber domain, in the phases of weapon deployment of an Autonomous Weapon System, results in a Comprehensive Human Oversight Framework depicted in Fig. 3. In this section we present the Comprehensive Human Oversight Framework by describing the layers and the connections between them and identifying gaps in the control mechanisms.

The Comprehensive Human Oversight Framework consists of three horizontal layers that are based on the three-layered model that Van den Berg (2015) created to describe cyber space. These layers can also be linked to the accountability mechanisms and control perspectives described, respectively in Sects. 2 and 3. On the x-axis time is plotted which can be divided into three phases: (1) before deployment of a weapon, (2) during deployment of a weapon and (3) after deployment of a weapon. These phases are depicted by the vertical columns of the framework. The y-axis describes the environment of the system which can range from more internal to more external to the technical system.

The combination of layers and columns result in nine blocks that each contain a component of control in each phase and layer. For example, before deployment the input to a system is a concept to control the goal of the system in the technical layer. The Comprehensive Human Oversight Framework allows to highlight the existence of gaps in control. These are presented below.

The figure above depicts the three layers of the Comprehensive Human Oversight Framework. The bottom technical layer describes the internal environment of the system and the upper governance layer the external environment of the system. The middle socio-technical layer is the intersection between the internal and external environment.

From a military perspective, control is described as a process to check if current and planned orders are on track and if the objectives to achieve a goal are met (Alberts and Hayes 2006; Liao 2008; NATO 2017). Control aims to make adjustments to the plan if the current state deviates from the planned end-state of the mission. Control measures bound the mission space by limiting the area of operation, duration of military operations and by defining the order of battle. Control consists of procedures for planning, directing and coordination of resources for a mission and this includes standard operating procedures (SOPs), rules of engagement (ROEs), regulations, military law, organizational structures and policies (Pigeau and McCann 2002). Control in a military perspective is an instrument to bound and check if the actions are in line with the planned military goal and to adjust the planning when the current state deviates from the end state. This resembles the notion of control in an engineering perspective because there is a goal, input and feedback loop to adjust the system.

In the military domain a variety of instruments are used as control mechanisms before, during and after deployment of weapons in military operations. After our analysis of the control mechanisms in the governance, socio-technical and technical perspectives on control in Sect. 3, we turned to the military domain to identify the military control instruments that are currently used in the three layers. We found that in the military domain there is a control mechanism in each layer before, during and after deployment of a weapon system. For each block examples of these mechanisms in the Netherlands are plotted in the Comprehensive Human Oversight Framework (Fig. 4) and described below.

The military control instruments in the Comprehensive Human Oversight Framework per block in the Netherlands are:

Contrary to the analysis of the academic literature describing the control mechanisms in the governance, socio-technical and technical perspective in Fig. 3, the military domain has an oversight mechanism during deployment in block 2 (see Fig. 4). The targeting process in block 2 is a decision-making process for methodical planning of actions to counter opponents in order to achieve the effect in the strategic and operational campaign plan (NATO 2016). The targeting process is a domain specific process for the military and is not monitored by an independent institution. By this, it is comparable to the statement of Pesch (2015) that an institutional structure for engineers is lacking to call on them to recognize, reflect upon and actively integrate values into the designs on a structural basis. Like engineers, the military does not have an independent institutional structure to call on them to reflect upon their values and principles during deployment. Reflection is done within the military domain and if military personnel violate military law and regulations they have to account for their conduct at a military court. However, this accountability process will be conducted after deployment and is not part of the targeting process during deployment.

The military control instruments in Fig. 4 are connected in the vertical columns of the layers. For example, the Rules-of-Engagement (block 4) will be based upon the Mission Mandate (block 1) and the options for weapon control status setting (block 7) will be determined by the Rules of Engagement (block 4). This is also the case for the horizontal levels as the Rules of Engagement (block 4) determine the guidelines of the Forward Air Control (block 5) and the After Action Report (block 6). This also applies to the bottom-up process after deployment. The Battle Damage Assessment (block 9) will be input for the After Action Report (block 6). The After Action Reports (block 6) should be used in the Post Mission Review process (block 3).

The feedback loop in the governance level from the Post Mission Review process (block 3) to the Mission Mandate (block 1) is often not conducted. A reason for this might be that different institutions are responsible for these instruments. The UN or NATO will draft the Mission Mandate and the Post Mission Review process is a national instrument. It is difficult to embed a national perspective in a multilateral document. In the socio-technical and technical level this feedback loop is conducted more often as these are within the military sphere of influence. For example, the Rules of Engagement (block 4) can be adjusted based on the findings of After Action Reports (block 6) and the Forward Air Control procedures (block 3) can be changed in accordance with the Rules Of Engagement (block 4). We recommend to try to close the feedback loop in the governance level so that findings in the Post Mission Review process will feed back into the Mission Mandate.

In the next section we apply the Comprehensive Human Oversight Framework to the case of Autonomous Weapon Systems. We describe the implications for the applicability of military control instruments for Autonomous Weapon Systems with different levels of autonomy. We compare the Comprehensive Human Oversight Framework presented in Sect. 4, which is based on the literature, to the application of the Framework to Autonomous Weapon Systems. This reveals two gaps in the control mechanisms that arise when the concept of autonomy is introduced in weapon systems which can be linked to the accountability gaps in Sect. 2.

The difference between a conventional weapon system and an Autonomous Weapon System is the notion of autonomy which is a not well-defined and often misunderstood concept. Castelfranchi and Falcone (2003) define autonomous as a notion that involves relationships between three entities: (a) the main subject (\(x\)), (b) the goal (\(\mu\)) that must be obtained by the main subject (\(x\)) and (c) a second subject (\(y\)) upon the main subject (\(x\)) is autonomous. This is expressed in the statement: “\(x\) is autonomous about \(\mu\) with respect to \(y\)”. For example, if (\(x\)) is an autonomous drone, its autonomy implies that the autonomous drone (\(x\)) can autonomously decide on the travel route (the goal (\(\mu\)) given a destination (i.e. GPS coordinates) set by its operator (\(y\)). Three type of autonomy relationships can be identified based on this description: (1) executive autonomy; (\(x\)) is autonomous in its means instead of it goals, which is the case of the example of the autonomous drone, (2) goal autonomy; (\(x\)) can set its goals on its own, and (3) social autonomy; (\(x\)) can execute its goals by itself without other agents (Castelfranchi and Falcone 2003).

Weapon systems may comprise of different levels of autonomy. But even in the case of a “fully Autonomous Weapon System”, ‘[…] that, without human intervention, selects and engages targets matching certain predefined criteria, following a human decision to deploy the weapon on the understanding that an attack, once launched, cannot be stopped by human intervention.’ (AIV and CAVV 2015) the type of autonomy can at most be executive autonomy, because a human will set its goals and the weapon will not decide on its goals or deployment itself. Also, the context will constrain the autonomy of a “fully Autonomous Weapon System” as autonomous systems are created with task goals and boundary conditions (Bradshaw et al. 2013). In case of Autonomous Weapon Systems, the context will include physical limitations to the area of operations, for example the presence, or lack of, civilians in the land, sea, cyber, air or space domain.

This notion of executive autonomy has implications for the applicability of military control instruments for Weapon Systems with different levels of autonomy, including fully Autonomous Weapon Systems. In the different phases executive autonomy implies that:

The current military control mechanisms described above are sufficient to bound the area of operation, the duration of the operation and deployment of weapons. But the introduction of autonomy in Autonomous Weapon Systems has implications on the military control mechanisms, mainly in the socio-technical layer during deployment of an Autonomous Weapon System. This may require reformation of the military control instruments. These implications might lead to new training methods for military personnel for them to have the capacity (knowledge and skills) to responsibly deploy these weapons, but might also lead to new institutions and design methods, for example value-sensitive design in (military) engineering (Santoni de Sio and Van den Hoven 2018), as control mechanisms in the governance layer.

Comparing the Comprehensive Human Oversight Framework in Fig. 3 to that of Autonomous Weapon Systems in Fig. 5 reveals two gaps in the control mechanisms that can be linked to the accountability gaps in Sect. 2: (1) a mechanism in block 2 of an independent institution that ensures oversight of a weapon during deployment (a governance accountability gap), and (2) in the Comprehensive Human Oversight Framework for Autonomous Weapon Systems there is no ongoing control mechanism in block 5 to control the specific actions that the Autonomous Weapon System takes to achieve its goal (a socio-technical accountability gap). On the one hand, fully executive autonomy inherently implies that the Autonomous Weapon System is autonomous in setting its means to achieve its goal independently from the human operator. On the other hand, even less-than-fully Autonomous Weapon Systems may still present big challenges in allowing the human controller to have effective control and supervision. This may actually depend, among other things, on the extent to which the ex ante and ex post mechanisms of control over the human–machine interaction are sufficient to give the operator the relevant capacities and motivation to discharge her duties. At a broader level, this arguably also depends on the extent that the governance level can provide an acceptable level of control on the choice of weapons and the distribution of tasks and duties in the mission.

The notions of accountability and responsibility are key concepts in the societal and academic debate. Accountability can be regarded as backward-looking responsibility to account for conduct after actions have occurred. However, many point to the fact that Autonomous Weapon Systems may lead to an accountability gap, accountability vacuum or an erosion of accountability relationships. We have identified three possible accountability gaps on three different layers: (1) a technical accountability gap, (2) a socio-technical accountability gap and a (3) a governance accountability gap. Accountability is a form of control and the notion of control can be viewed from different perspectives. In this paper we mention the engineering perspective, the socio-technical perspective and the governance perspective. Our main claim is that combining the control mechanisms in the technical, socio-technical and governance layer will lead to Comprehensive Human Oversight over Autonomous Weapon Systems which may ensure solid controllability and accountability for the behaviour of Autonomous Weapon Systems.

These three perspectives on control constitute the three layers of our proposed Comprehensive Human Oversight Framework. The Comprehensive Human Oversight Framework highlights the connection between the layers and shows an existing gap in the governance layer. Current military control instruments cover the blocks of the Comprehensive Human Oversight Framework. However, when applied to Autonomous Weapon Systems the Comprehensive Human Oversight Framework reveals two gaps in control, one gap in the governance layer and one in the socio-technical layer. Future work will have to address these gaps to assess whether other gaps may emerge at other levels which need to be filled in order to ensure accountability over Autonomous Weapon Systems.