Safety Demonstration Framework of Automated Road Mobility: Methodological and Governance Aspects of the Scenario-Based Approach

Automated driving systems (ADS) with high levels of automation require setting commonly and widely accepted and applied high-level safety rules and efficient validation framework to ensure safety.

Within the UNECE World Forum for Harmonization of Vehicle Regulations (WP.29), a dedicated group under Working Party on Automated/Autonomous and Connected Vehicles (GRVA) is working on safety validation tools and methods (VMAD). Despite the path to regulation has not been done yet, the informal working group delivered their first document in June 2021 as a reference on safety validation methods and tools for vehicle type-approval [1]. This first publication at the UN level opens the global approach that national and regional authorities will have to follow. In particular, the Master document on New Assessment and Test Methods (NATM) considers six pillars to demonstrate safety of a vehicle, among which is the scenario catalogue. By providing the foundations of a robust safety validation framework in a repeatable, objective, evidence-based and technology neutral manner; the scenario-based approach is defined as to be the starting point to help to systematically organize safety validation activities in such a way. Moreover, the question of ensuring a holistic and dense coverage of traffic situations is central. Finally yet importantly, NATM guidelines closely associate two concepts: scenarios and operational design domain (ODD), in which the system is designed to operate safely. The effectiveness of a scenario-based approach is presented through the correspondence between the scenario catalogue and ODD real-world driving situations that can reasonably be expected to occur in the ODD.

On this basis the need to have international standards gathering and merging international knowledge on scenarios is pushed by the international community of experts. The aim of establishing both international and national standards on scenarios lies in the need of harmonizing practices and considerations around what is considered as one of the most important topic to address regarding safety but also one of the most difficult to properly address regarding the variety of systems and ODDs. In this context, an international standard is developing at ISO under the TC22/SC33 committee. The project’s scope under discussions is composed of a set of five standards on scenarios [2].

At the European Union (EU) level, the challenge of regulating the safe operation of automated driving systems (ADS) has been made. The European Commission (EC) published last year the first international regulation 2022/1426 on ADS for vehicle type-approval [3]. This regulation relies heavily on scenarios that are relevant for the system. Scenario generation is described as it should cover “reasonably foreseeable” traffic situations and it goes further by considering that scenario parameter should not be limited to values already observed in documented databases.

The national doctrine on automated road mobility is supported by the national strategy, revised in 2022 for 2025 [4]. Since 2018, France has built its framework on three principles – safety, progressivity, and acceptability. These three pillars are intended to both preserve road users and vehicles’ integrity and promote confidence in automated driving systems, being aware that the more people trust, the more they use. To fulfil this aim it appeared relevant to rely on the strength of a complete legislative and regulatory framework.

The French regulatory framework on automated mobility entered into force in September 2022. The legislative basis of automated road mobility in France comes from the “Loi d’orientation des mobilités” (LOM) of December 24, 2019, and Ordinance no. 2021–443 from April 14, 2021 which define roles and responsibilities of automated systems and drivers or human supervisors [5]. The regulatory package for the deployment of automated road transport systems is composed of Decree no. 2021–873 from June 29, 2021, which defines a global process to demonstrate safety, putting safety demonstration at the center of the loop, which leads to consider the foundation of a performance- and progressive-based approach. Such an approach is reinforced by recent academic work on demonstrating safety of automated vehicles, on which it is possible to rely [6].

The interest of the scenario-based approach, within the national context, relies on three main, complementary logics (a) a logic of covering the space of possibilities, which meets the objective of not having omitted cases, characterizing the search of representativity; (b) an essentially probabilistic logic regarding their potential (very) high criticality, giving life to the concept of “edge-cases”; and (c) a careful and restrictive logic, considering a set of driving situations in which the (static) environment gives possible addressable hazards characterized with complex interactions.

The scenario-based approach, among safety demonstration activities regarding automated road transport systems’ deployment, aims to address the relevant level of details for an ADS depending on its capabilities, while remaining technology neutral. Indeed assessing safety of a specific system requires sizing precisely the set of scenarios by combining approaches focusing respectively on the vehicle, the system in which it will be integrated, the route on which it will be deployed (at a local level). That is to say, the scenario-based approach has the virtue to proportionate safety demonstration to the ODD, including ODD boundaries.

Based on NATM requirements [1], working on scenario generation to assess vehicle safety, and more generally system safety constitutes a pillar that is (a) more precise, in particular by taking into account ODD limits attached to the use cases; (b) more complete, in particular by using risk analysis to fill the gaps in traffic hazards analysis; (c) more diverse, in particular through the accumulation of feedbacks or the addition of other scenario description axes as the use cases mature (e.g. remote intervention, interactions with law enforcement officers and/or first responders).

The scenario-based approach being at the core of safety demonstration, defining clear and common principles on the methodology is a key element. A national doctrine has been launched, based on a guiding principle linked to the notion of “reasonably foreseeable”. The reasonably foreseeable concept has to be seen as the notion of what is conceivable, in the sense that any situation that might occur within the entire ODD has to be taken into account. The contribution relies on defining the reasonably foreseeable principle as referring to a global safety demonstration principle related to scenario generation, regardless of the type of scenarios and independently of the scenario selection process, to provide an “exhaustive” coverage of traffic situations that might occur and that are conceivable for the specific system in its ODD.

Based on a large literature review on scenario-based approaches [7‐9] that emerged at the international level through projects, work showed the potential and, at the same time, complexity of multiplying axes of scenario description.

The question of the opportunity of standardizing scenario description is complex. On the one hand, considering that, the scenario approach is subsidiary to the description of ODDs and should aim at the maximum exhaustiveness of possible scenario descriptors, the standardization of descriptors may seem inappropriate. On the other hand, the idea that scenario generation is a cumulative process of expertise and/or data makes the standardization of descriptors meaningful.

It is assumed that the proposed doctrine and method presented here, shared and supported by national industry is one possible logical arrangement. The proposed scenario-based approach is based on the combination of five layers (Table 1):

The strength of this approach is to be able to answer the principle of scenario coverage, by avoiding omitting certain types of events. The power of such an approach is accountable to the power to combine these axes and to precise them through attributes. The resulting living process amounts to saying that working on scenario generation is worth working on scenario descriptors.

The combinatory-based approach pushes to consider the two-level loop along scenarios: managing the scenario catalogue from both knowledge- and data-based methods, supplemented by creating new combinations from descriptor combination itself (Fig. 1).

Bild vergrößern

The notion of scenario completeness alone might lead to consider an infinite number of scenarios. In that perspective, the concept of representativity plays a great role, especially in conjunction with ODD consideration. The scenario approach should also address the question how to move from ODD and OED(R) to scenarios that give life to a set of descriptive bricks. The search for completeness in the lists of descriptors is the key to move from inert elements to sets of characteristics intended to build a scenario.

French approach based on scenario generation and on completeness does not seek to describe neither characterize triggering conditions of hazards in a specific ODD while it may freeze the ODD itself and reasonably foreseeable situations linked to it. Otherwise, it aims at clarifying driving situations translated in scenarios related to an ODD and addressable hazards relevant in its ODD. Having saying that, creating scenarios on driving situations based on a specific ODD, regarding possible addressable events, means having to consider ODD description and OED definition themselves.

Bild vergrößern

Based on that, there is a need to build a scenario methodology to move forward from a possible infinite (functional) scenario generation process to its intersection with ODD description and OED(R) definition, enabling to qualify, classify and prioritize logical to concrete scenarios to be included in testing procedures. In that case, it is worth mentioning that ODD consideration through its descriptors has to be linked with OED descriptors (see Fig. 2) as they constitute a set of the reasonably foreseeable conditions. As a result, all these considerations place ODD has a central element when building a scenario generation process, and put in the middle the concept of descriptors.

This ties in with the idea that the search of completeness within the scenario-based approach, carried out by the notion of a sufficient list of descriptors is the key to move from inert elements to sets of characteristics intended to build a scenario. Hence, making the link between these two important notions is a possible way to move from a theoretical approach named scenario generation to a concrete approach characterized by its intersection with ODD description. The latter process aims to provide a finite number of concrete scenarios for testing procedures.

Moving from scenario generation, based on the combination of descriptors that might lead to a non-quantifiable number of scenarios, to scenario testing that means a fully discretization of the space of scenarios, reflects moving from a generic ODD to a specific ODD (called operational domain (OD)), instantiated by hazards and events.

Starting from bounded, discretized space of ODD descriptors, the approach lies on the principle that building a finite list of scenarios is possible. The logic of completeness, translated in the notion of scenario coverage focuses on three major areas: (a) to bring out the so-called unavoidable or generic scenarios, which by their frequency are representative of road traffic and existing situations, and which demonstrate a potential severity that justifies an interest regarding safety; (b) to target edge case scenarios, first at the critical axis limits, generally characterized by an extreme severity, then designed from unknown descriptors; and, (c) with the idea of reflecting cautiousness behaviors.