Skip to main content

2019 | OriginalPaper | Buchkapitel

Uncertainty Wrappers for Data-Driven Models

Increase the Transparency of AI/ML-Based Models Through Enrichment with Dependable Situation-Aware Uncertainty Estimates

verfasst von : Michael Kläs, Lena Sembach

Erschienen in: Computer Safety, Reliability, and Security

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

In contrast to established safety-critical software components, we can neither prove nor assume that the outcomes of components containing models based on artificial intelligence (AI) or machine learning (ML) will be correct in any situation. Thus, uncertainty is an inherent part of decision-making when using the outcomes of data-driven models created by AI/ML algorithms. In order to deal with this – especially in the context of safety-related systems – we need to make uncertainty transparent via dependable statistical statements. This paper introduces both a conceptual model and the related mathematical foundation of an uncertainty wrapper solution for data-driven models. The wrapper enriches existing data-driven models such as provided by ML or other AI techniques with case-individual and sound uncertainty estimates. The task of traffic sign recognition is used to illustrate the approach, which considers uncertainty not only in terms of model fit but also in terms of data quality and scope compliance.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Fußnoten
1
Since we cannot obtain representative samples for all \( case TAS \), we make a worst-case approximation by assuming \( p (m\left( X \right) = O | caseTAS) = 0 \), i.e., outcomes are never correct.
 
Literatur
1.
Zurück zum Zitat Solomatine, D., Ostfeld, A.: Data-driven modelling: some past experiences and new approaches. J. Hydroinform. 10(2), 3–22 (2008)CrossRef Solomatine, D., Ostfeld, A.: Data-driven modelling: some past experiences and new approaches. J. Hydroinform. 10(2), 3–22 (2008)CrossRef
3.
Zurück zum Zitat Kläs, M., Vollmer, A.M.: Uncertainty in machine learning applications – a practice-driven classification of uncertainty. In: First International Workshop on Artificial Intelligence Safety Engineering (WAISE 2018) (2018) Kläs, M., Vollmer, A.M.: Uncertainty in machine learning applications – a practice-driven classification of uncertainty. In: First International Workshop on Artificial Intelligence Safety Engineering (WAISE 2018) (2018)
4.
Zurück zum Zitat Armstrong, J.S.: The Forecasting Dictionary. In: Principles of Forecasting: A Handbook for Researchers and Practitioners, Springer Science & Business Media (2001) Armstrong, J.S.: The Forecasting Dictionary. In: Principles of Forecasting: A Handbook for Researchers and Practitioners, Springer Science & Business Media (2001)
5.
Zurück zum Zitat Kläs, M.: Towards identifying and managing sources of uncertainty in AI and machine learning models - an overview. arXiv preprint arXiv:1811.11669 (2018) Kläs, M.: Towards identifying and managing sources of uncertainty in AI and machine learning models - an overview. arXiv preprint arXiv:​1811.​11669 (2018)
6.
Zurück zum Zitat Lee, S., Chen, W.: A comparative study of uncertainty propagation methods for black-box-type problems. Struct. Multidiscip. Optim. 37, 239 (2009)CrossRef Lee, S., Chen, W.: A comparative study of uncertainty propagation methods for black-box-type problems. Struct. Multidiscip. Optim. 37, 239 (2009)CrossRef
7.
Zurück zum Zitat Safavian, S., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Trans. Syst. Man Cybern. B Cybern. 21(3), 660–674 (1991)MathSciNetCrossRef Safavian, S., Landgrebe, D.: A survey of decision tree classifier methodology. IEEE Trans. Syst. Man Cybern. B Cybern. 21(3), 660–674 (1991)MathSciNetCrossRef
8.
Zurück zum Zitat Khosravi, A., Nahavandi, S., Creighton, D., Atiya, A.: Comprehensive review of neural network-based prediction intervals and new advances. IEEE Trans. Neural Networks 22(9), 1341–1356 (2011)CrossRef Khosravi, A., Nahavandi, S., Creighton, D., Atiya, A.: Comprehensive review of neural network-based prediction intervals and new advances. IEEE Trans. Neural Networks 22(9), 1341–1356 (2011)CrossRef
9.
Zurück zum Zitat Gal, Y.: Uncertainty in Deep Learning, University of Cambridge (2016) Gal, Y.: Uncertainty in Deep Learning, University of Cambridge (2016)
10.
Zurück zum Zitat McAllister, R., et al.: Concrete problems for autonomous vehicle safety: advantages of Bayesian deep learning, In: International Joint Conferences on Artificial Intelligence (2017) McAllister, R., et al.: Concrete problems for autonomous vehicle safety: advantages of Bayesian deep learning, In: International Joint Conferences on Artificial Intelligence (2017)
11.
Zurück zum Zitat Kläs, M., Trendowicz, A., Wickenkamp, A., Münch, J., Kikuchi, N., Ishigai, Y.: The use of simulation techniques for hybrid software cost estimation and risk analysis. Adv. Comput. 74, 115–174 (2008)CrossRef Kläs, M., Trendowicz, A., Wickenkamp, A., Münch, J., Kikuchi, N., Ishigai, Y.: The use of simulation techniques for hybrid software cost estimation and risk analysis. Adv. Comput. 74, 115–174 (2008)CrossRef
12.
Zurück zum Zitat Angelis, L., Stamelos, I.: A simulation tool for efficient analogy based cost estimation. Empir. Softw. Eng. 5(1), 35–68 (2000)CrossRef Angelis, L., Stamelos, I.: A simulation tool for efficient analogy based cost estimation. Empir. Softw. Eng. 5(1), 35–68 (2000)CrossRef
13.
Zurück zum Zitat Shrestha, D., Solomatine, D.: Machine learning approaches for estimation of prediction interval for the model output. Neural Netw. 19(2), 225–235 (2006)CrossRef Shrestha, D., Solomatine, D.: Machine learning approaches for estimation of prediction interval for the model output. Neural Netw. 19(2), 225–235 (2006)CrossRef
14.
Zurück zum Zitat Solomatine, D., Shrestha, D.: A novel method to estimate model uncertainty using machine learning techniques. Water Resour. Res. 45(12), W00B11 (2009)CrossRef Solomatine, D., Shrestha, D.: A novel method to estimate model uncertainty using machine learning techniques. Water Resour. Res. 45(12), W00B11 (2009)CrossRef
15.
Zurück zum Zitat Brown, L., Cai, T., DasGupta, A.: Interval Estimation for a Binomial Proportion. Stat. Sci. 16(2), 101–133 (2001)MathSciNetMATH Brown, L., Cai, T., DasGupta, A.: Interval Estimation for a Binomial Proportion. Stat. Sci. 16(2), 101–133 (2001)MathSciNetMATH
Metadaten
Titel
Uncertainty Wrappers for Data-Driven Models
verfasst von
Michael Kläs
Lena Sembach
Copyright-Jahr
2019
DOI
https://doi.org/10.1007/978-3-030-26250-1_29

Premium Partner