Trustworthy AI on the Road

The AI HLEG states that ‘Trustworthiness is a prerequisite for people and societies to develop, deploy and use AI systems’ (AI HLEG 2019, 4). When not worthy of trust, unwanted consequences could develop and the uptake of AI systems might be hindered (AI HLEG 2019, 4). The experts list three requirements that should be met throughout the lifetime of an AI-system: it should be lawful, ethical and robust (AI HLEG 2019, 5). The AI HLEG develops principles that contribute to the requirements of robust and ethical AI, as they ‘proceed on the assumption that all legal rights and obligations that apply to the processes and activities involved in developing, deploying and using AI systems remain mandatory and must be duly observed’ (AI HLEG 2019, 6). Four ethical principles are formulated: (1) respect for human autonomy, (2) prevention of harm, (3) fairness, (4) explicability (AI HLEG 2019, 12). The AI HLEG expert group derives 7 requirements from these principles (though this list is not exhaustive) (AI HLEG 2019, 14): (1) Human agency and oversight; (2) Technical robustness and safety; (3) Privacy and data governance; (4) Transparency; (5) Diversity, non-discrimination and fairness; (6) Societal and environmental wellbeing; (7) Accountability. Compliance with these principles should ensure the trustworthiness of AI, thereby facilitating the safe and responsible development of AI systems.

Whereas the principles of the AI HLEG are designed for trustworthy AI in general, the Horizon 2020 Expert Group focuses specifically on connected and automated vehicles (CAVs). The Horizon 2020 Expert Group has drawn up 20 recommendations that can be divided into three broader topics/themes: (1) road safety, risk dilemmas, (2) Data and algorithm ethics: privacy, fairness and explainability, (3) responsibility (Horizon 2020 Expert Group 2020, 4–5). The recommendations range from reducing physical harm (recommendation 1) to safeguarding informational privacy (recommendation 5) to creating fair and effective mechanisms for granting compensation to victims of crashes or other accidents involving CAVs (recommendation 20). It goes beyond the scope of this contribution to discuss the recommendations more in-depth, but it is already clear that several recommendations of the Horizon 2020 Expert Group show similarities to the 7 requirements formulated by the AI HLEG.

Ensuring the technical robustness and safety of automated vehicles can be achieved via different routes, including the EU type-approval framework as laid down in the EU Type-approval Regulation. The EU General Safety Regulation already sets requirements for (fully) automated vehicles in art. 11. In addition, requirements from the UN Regulations (can) contribute to the automated vehicle’s safety and technical robustness. UN R155 on cybersecurity, for instance, is an important instrument in ensuring vehicles’ digital security and thereby fostering the safety of vehicles.

Privacy and data governance are not just important in relation to the automotive field, they are topics that are of great importance in all sectors. Regardless of the sector, when it comes to processing personal data (‘‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person’) the General Data Protection Regulation (GDPR, art. 1, 4) applies. In addition, the EU Data Governance Act sets requirements for the re-use, within the EU, of certain categories of data held by public sector bodies (art. 1) and the proposed Data Act lays down harmonised rules on making data generated by the use of a product (like an automated vehicle) ‘available to the user of that product or service, on the making data available by data holders to data recipients, and on the making data available by data holders to public sector bodies or Union institutions, agencies or bodies, where there is an exceptional need, for the performance of a task carried out in the public interest’ (art. 1(1)). Legislation specifically for (automated) vehicles also lays down some rules on the recording of data: the General Safety Regulation provides some provisions on data recording (e.g. art. 6(4)), as work is being done on a data storage system for automated driving (DSSAD) at a UN level.

Other than with robustness and safety as well as privacy and data governance, transparency seems to be lacking in the current legal framework for automated vehicles. There are elements of transparency in UN regulations and the type-approval process, but these requirements do not seem to achieve the level of transparency strived for in the proposed AI Act: ‘High-risk AI systems shall be designed and developed in such a way to ensure that their operation is sufficiently transparent to enable users to interpret the system’s output and use it appropriately’ (art. 13(1)). Especially when it comes to transparency towards users, legal lacunas seem to exist. Domestic legislation could fill this gap, setting requirements in this. In addition, the EU Product Liability Directive might provide an incentive for manufacturers of automated vehicles to inform the vehicles’ users of its possibilities and limitations: if not or incorrectly informed, the manufacturer might be found liable for resulting damage (art. 6(1)).

The proposed AI Act requires that ‘High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use’ (art. 14(1)). Human oversight, according to the AI HLEG, can help in ensuring human autonomy by for instance enabling human intervention in every decision cycle of the system (AI HLEG, 16). This gives rise to the question of whether human oversight is contradicting the idea of automated vehicles, in which a human is taken ‘out-of-the-loop’ and does not take part in the decision making concerning the execution of the driving task. This could explain why the Horizon 2020 Expert Group does not refer to human oversight in this meaning. Regarding human oversight, this expert group does however explain that algorithmic decisions require adequate explaination (which is closely tied to the requirement of transparency) because ‘Without adequate means of access, the role of human agency and oversight is severely weakened or hindered and risks undermining the principles of human dignity and autonomy, with the consequence of critically eroding public trust in these fastdeveloping technologies’ (Horizon 2020 Expert Group, 50). The AI HLEG adds to this that ‘the less oversight a human can exercise over an AI system, the more extensive testing and stricter governance is required’ (AI HLEG, 16). This could mean that establishing human oversight in the development phase might be sufficient. The UN regulations and EU Type-approval Regulation could ensure this.