Top

Published in:

2024 | OriginalPaper | Chapter

STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems

Authors : Till Schallau, Dominik Mäckel, Stefan Naujokat, Falk Howar

Published in: Dependable Computing – EDCC 2024 Workshops

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Extensive testing and simulation in different environments has been suggested as one piece of evidence for the safety of autonomous systems, e.g., in the automotive domain. To enable statements on the absolute number or fractions of tested scenarios, methods and tools for computing their coverage are needed. In this paper, we present STARS, a tool for specifying semantic environment features and measuring scenario coverage when testing autonomous systems.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Towards Continuous Assurance Case Creation for ADS with the Evidential Tool Bus

next chapter Spatial-Temporal Graph Neural Network for Detecting and Localizing Anomalies in PMU Networks

https://github.com/tudo-aqua/stars.

Note: The data model used for the evaluation of the TSCs is described in [17].

Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Standard J3016, SAE International (2021). https://www.sae.org/standards/content/j3016_202104/

Standard for safety for the evaluation of autonomous products. Standard ANSI/UL 4600-2023, UL Standards & Engagement (2023). https://ul.org/UL4600

Amersbach, C., Winner, H.: Defining required and feasible test coverage for scenario-based validation of highly automated vehicles. In: ITSC 2019, IEEE (2019). https://doi.org/10.1109/itsc.2019.8917534

Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: PMLR 2017, vol. 78. PMLR (2017). https://proceedings.mlr.press/v78/dosovitskiy17a.html

Duret-Lutz, A., Lewkowicz, A., Fauchille, A., Michaud, T., Renault, É., Xu, L.: Spot 2.0 — a framework for LTL and \(\omega \)-automata manipulation. In: Artho, C., Legay, A., Peled, D. (eds.) ATVA 2016. LNCS, vol. 9938, pp. 122–129. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46520-3_8CrossRef

Esterle, K., Gressenbuch, L., Knoll, A.C.: Formalizing traffic rules for machine interpretability. In: CAVS 2020, IEEE (2020). https://doi.org/10.1109/CAVS51000.2020.9334599

Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The KITTI dataset. Int. J. Robot. Res. (IJRR) 32(11), 1231–1237 (2013). https://doi.org/10.1177/0278364913491297CrossRef

Hildebrandt, C., von Stein, M., Elbaum, S.: PhysCov: physical test coverage for autonomous vehicles. In: ISSTA 2023, ACM (2023). https://doi.org/10.1145/3597926.3598069

Huck, T.P., Ledermann, C., Kröger, T.: Simulation-based testing for early safety-validation of robot systems. In: SPCE 2020, IEEE (2020). https://doi.org/10.1109/SPCE50045.2020.9296157

10.

ISO central secretary: road vehicles - safety of the intended functionality. Standard ISO 21448:2022, International Organization for Standardization (2022). https://www.iso.org/standard/77490.html

11.

Jenkins, I.R., Gee, L.O., Knauss, A., Yin, H., Schroeder, J.: Accident scenario generation with recurrent neural networks. In: ITSC 2018, IEEE (2018). https://doi.org/10.1109/itsc.2018.8569661

12.

Li, C., Cheng, C.H., Sun, T., Chen, Y., Yan, R.: ComOpT: combination and optimization for testing autonomous driving systems. In: ICRA 2022, IEEE (2022). https://doi.org/10.1109/icra46639.2022.9811794

13.

Majumdar, R., Mathur, A., Pirron, M., Stegner, L., Zufferey, D.: Paracosm: a language and tool for testing autonomous driving systems. Technical Report arXiv:1902.01084, arXiv (2021). https://doi.org/10.48550/arXiv.1902.01084

14.

Mariani, R.: An overview of autonomous vehicles safety. In: IRPS 2018, IEEE (2018). https://doi.org/10.1109/irps.2018.8353618

15.

Porres, I., Azimi, S., Lilius, J.: Scenario-based testing of a ship collision avoidance system. In: SEAA 2020, IEEE (2020). https://doi.org/10.1109/SEAA51224.2020.00090

16.

Rizaldi, A., et al.: Formalising and monitoring traffic rules for autonomous vehicles in Isabelle/HOL. In: Polikarpova, N., Schneider, S. (eds.) IFM 2017. LNCS, vol. 10510, pp. 50–66. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66845-1_4CrossRef

17.

Schallau, T., Naujokat, S.: Validating behavioral requirements, conditions, and rules of autonomous systems with scenario-based testing. Electron Communications of the EASST 82 (2023). https://doi.org/10.14279/tuj.eceasst.82.1222

18.

Schallau, T., Naujokat, S., Kullmann, F., Howar, F.: Tree-based scenario classification: a formal framework for coverage analysis on test drives of autonomous vehicles. Technical Report arXiv:2307.05106, arXiv (2023). https://doi.org/10.48550/arXiv.2307.05106, submitted to NFM 2024

19.

Schmidt, T., Hauer, F., Pretschner, A.: Understanding safety for unmanned aerial vehicles in urban environments. In: IV 2021, IEEE (2021). https://doi.org/10.1109/IV48863.2021.9575755

20.

Weber, H., et al.: A framework for definition of logical scenarios for safety assurance of automated driving. Traffic Injury Prev. 20(sup1) (2019). https://doi.org/10.1080/15389588.2019.1630827

21.

Weng, B., Capito, L., Ozguner, U., Redmill, K.: Towards guaranteed safety assurance of automated driving systems with scenario sampling: an invariant set perspective. IEEE Trans. Intell. Veh. 7(3) (2022). https://doi.org/10.1109/tiv.2021.3117049

Title: STARS: A Tool for Measuring Scenario Coverage When Testing Autonomous Robotic Systems
Authors: Till Schallau
Dominik Mäckel
Stefan Naujokat
Falk Howar
Publisher: Springer Nature Switzerland
Book: Dependable Computing – EDCC 2024 Workshops
Print ISBN: 978-3-031-56775-9

Electronic ISBN: 978-3-031-56776-6

Copyright Year: 2024
DOI: https://doi.org/10.1007/978-3-031-56776-6_6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner