Collective Perception Virtual Safety Validation in Urban Environments: Scenarios, Tools, Metrics

The on-board sensors of connected and automated vehicles (CAVs) are limited by their range and inability to see around corners or blind spots [1]. Collective (or cooperative) perception (CP) is a multi-agent system [2] in which agents share perceptual information such as their state (e.g., vehicle position, pose, speed, acceleration), their tracked object list or even their tracked objects’ intentions. CP is currently standardized by the European Telecommunications Standards Institute (ETSI) as a second-generation Vehicle-to-Everything (V2X) communication service. In that context, the CP testing has been studied more as an instantiation of a Vehicular Ad-hoc NETwork (VANET), via a combination of traffic/network simulation environments and targeting at answering questions such as up to how many connected agents per topology, communication protocols, latency and frequency of messages to be broadcasted, without studying the collective perception content generation (e.g. fusion techniques) and the assessment of the derived (collective) object detection.

Based on the above, the problem is cast as a multi-agent perception testing problem where the focus is on the shared information content (pre-processed sensor data) and how this is affecting (i.e. what sensory data needs to be shared, at which resolution, etc.) the CP information fusion on the CAV side under the presence of occlusions and sensor measurement uncertainties (uncertainty propagation). CAV control layer, i.e. which cooperative strategies should be implemented are out of scope (for a broader specification of cooperative driving automation goals, see SAE J3216), since the present work focuses only on the perception layer.

Following this trend, the goal of the present analysis is to explore requirements towards building a virtual safety validation framework for urban CP testing, concretely addressing current gaps with respect to (a) metrics for assessing CP from a perception point of view when late fusion is employed, i.e. where detected bounding boxes and confidence scores are shared – following the ETSI CPM formulation, (b) interesting scenarios for (a), and (c) which types of simulation tools are necessary for (a).

Several surveys of highly qualified content have been released in the last two years trying to cover the CP task research and development challenges, proving the growing interest of both automated driving (AD) perception and VANETs communities [3‐5]. In parallel, two surveys on object-level perception and CP safety assessment [6, 7] have also appeared, inspired by the recent ISO work on SOTIF [8]. Reviewing the aforementioned surveys and some of the works they are including, in the following sub-sections, we highlight certain aspects relevant to our CP testing context.

In recent years, there are many efforts on introducing large-scale CP datasets, for the automotive context, which are mostly generated by simulation and are appropriate for CP modules testing. This advent is partially due to the introduction of and community support for CARLA open source driving simulator [9] which makes easy the generation of virtual sensor data and offers a variety of ground truth data (incl. Instance semantic segmentation data that yields a unique pixel value for every object in a scene and pedestrian skeleton data). In July 2018, version 0.9.0 introduced the multi-client multi-agent support that opened the road for cooperative agents. CARLA supports basic sensor modeling for several sensors such as cameras, depth cameras, LiDAR (simulated ray cast), IMU and RADAR.

CP datasets generated in simulation: The first big contribution was made in 2021 by UCLA Mobility lab with the release of OpenCDA dataset and simulation benchmark [10] which supported testing both on individual autonomy level and traffic level and built a co-simulation platform (CARLA is part of it), a full-stack prototype cooperative driving system, and a scenario manager. OpenCDA also offered benchmark testing scenarios, state-of-the-art benchmark algorithms for all modules of an AD stack, benchmark testing road maps, and benchmark evaluation metrics but focusing on cooperative driving applications and not on CP. It is also noted that type of data exchanged among traffic agents are freely explored and not restricted to the messages already standardized by ETSI. OpenCDA has grown since then and today it constitutes an Open-source Ecosystem for Cooperative Driving Automation Research with an active community developing part of it around the globe [11]. Two similar efforts followed focusing on multi-agent perception: OPV2V [12] and V2X-Sim [13]. Both datasets provide object-level annotations from CARLA towns in order to support detection, tracking and segmentation perception tasks deploying early fusion/feature-fusion/late-fusion approached for CP content generation. OPV2V is implemented by the COOD framework which supports multi-GPU training and is used by V2VSet [14]. Scenarios supported by all the simulation frameworks above focus on urban, rural and more seldom on highway operational domain and include V2X in straight and curvy urban road segments, urban intersections, rural areas where only V2V applies and highway on-ramp scenarios. The aforementioned CP simulation benchmarks are equipped with RGB cameras and Lidar, allowing the collection of more than 10,000 frames and each scene contains at least 2 vehicles. As discussed in V2XViT [14], in simulation different perception error models can be introduced (pose error, agents’ synchronization error, time delay in V2X communication).

CP datasets from real-world recordings (incl. Infrastructure data): DAIR-V2X, OpenDAIR-V2X and soon to be released V2X-Seq [15] are the first real-world datasets for research on V2X AD. It comprises image data and LiDAR point cloud data from different observers. Notably, it supports early fusion and late fusion CP methods while it is planned to also support feature fusion.

Notably, in the initial OpenCDA benchmark, metrics for CAD safety assessment are proposed but metrics focused on CP are missing. Metrics for perception or CP based on sensor data late fusion methods are provided in OpV2V, V2XSim and V2XViT [14] depending on the CP algorithms in use. Still, as argued in [6] metrics and criteria that explicitly distinguish safety-relevant from safety-irrelevant perception errors are still under-explored.

Bild vergrößern

Starting with the three core subsystems, namely the task manager, the simulator and the simulation model validation, identified in the SUNRISE harmonized V&V simulation framework conceptual architecture [16], the System-under-Test (SuT) and problem formulation for the CP assessment task are discussed hereafter with the help of the diagram of Fig. 1.

In our setting, SuT is the local or distributed CP subsystem of the CAV under test as part of two possible CP system configurations, inline with [17]:

The goal of this section is to propose high-level validation requirements for urban CP scenario-based testing.

In the present work collective perception validation challenges were discussed in detail based on comprehensive SoA review. Moreover, a preliminary set of high-level safety validation requirements for collective perception scenario-based testing in simulation is derived, by focusing on urban environments.

Future work will include application of these requirements for assessing a Bayesian-based CP module supporting virtual agents that can exchange ETSI CPM information and corresponding CP system safety validation in CARLA.