nach oben

Empirical Software Engineering

Erschienen in:

01.06.2023

VEER: enhancing the interpretability of model-based optimizations

verfasst von: Kewen Peng, Christian Kaltenecker, Norbert Siegmund, Sven Apel, Tim Menzies

Erschienen in: Empirical Software Engineering | Ausgabe 3/2023

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Context:

Many software systems can be tuned for multiple objectives (e.g., faster runtime, less required memory, less network traffic or energy consumption, etc.). Such systems can suffer from “disagreement” where different models have different (or even opposite) insights and tactics on how to optimize a system. For configuration problems, we show that (a) model disagreement is rampant; yet (b) prior to this paper, it has barely been explored.

Objective:

We aim at helping practitioners and researchers better solve multi-objective configuration optimization problems, by resolving model disagreement.

Method:

We propose a dimension reduction method called VEER that builds a useful one-dimensional approximation to the original N-objective space. Traditional model-based optimizers use Pareto search to locate Pareto-optimal solutions to a multi-objective problem, which is computationally heavy on large-scale systems. VEER builds a surrogate that can replace the Pareto sorting step after deployment.

Results:

Compared to the prior state-of-the-art, for 11 configurable systems, VEER significantly reduces disagreement and execution time, without compromising the optimization performance in most cases. For our largest problem (with tens of thousands of possible configurations), optimizing with VEER finds as good or better optimizations with zero model disagreements, three orders of magnitude faster.

Conclusion:

When employing model-based optimizers for multi-objective optimization, we recommend to apply VEER, which not only improves the execution time, but also resolves the potential model disagreement problem.

Vorheriger Artikel Towards a change taxonomy for machine learning pipelines

Nächster Artikel Bugs in machine learning-based systems: a faultload benchmark

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

http://tiny.cc/gartners21

Holland’s advice (Holland 1992) for genetic algorithms (such as NSGA-II and MOEA/D) is that 100 individuals need to be evolved over 100 generations; i.e., 10⁴ evaluations in all.

And the source code for that implementation can be found in the reproduction package mentioned in our abstract

We define a concordant pair in tasks of more than 2 objectives in a similar manner: one configuration has better performance than the other in all objectives. This is not originally defined by Kendall, but we believe it is a proper extension.

Agrawal A, Menzies T, Minku LL, Wagner M, Yu Z (2020) Better software analytics via “duo”: data mining algorithms using/used-by optimizers. Empir Softw Eng 25(3):2099–2136CrossRef

Antkiewicz M, Bąk K, Murashkin A, Olaechea R, Liang JH, Czarnecki K (2013) Clafer tools for product line engineering. In: Proceedings of the 17th international software product line conference co-located workshops, pp 130–135

Bergstra J, Bardenet R, Bengio Y, Kégl B (2011) Algorithms for hyper-parameter optimization. Adv Neural Inf Process Syst, vol 24

Bergstra J, Yamins D, Cox DD et al (2013) Hyperopt: a python library for optimizing the hyperparameters of machine learning algorithms. In: Proceedings of the 12th Python in science conference, Citeseer, vol 13, p 20

Brochu E, Cora VM, De Freitas N (2010) A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:10122599

Chen D, Fu W, Krishna R, Menzies T (2018a) Applications of psychological science for actionable analytics. In: Proceedings of the 2018 26th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 456–467

Chen J, Nair V, Krishna R, Menzies T (2018b) “sampling” as a baseline optimizer for search-based software engineering. IEEE Trans Softw Eng (pre-print):1–1

Coello CAC, Sierra MR (2004) A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm. In: Mexican international conference on artificial intelligence. Springer, pp 688–697

Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evolution Computat 6(2):182–197CrossRef

Devanbu P, Zimmermann T, Bird C (2016) Belief & evidence in empirical software engineering. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 108–119

Gigerenzer G (2008) Why heuristics work. Perspect Psycho Sci 3 (1):20–29CrossRef

Golovin D, Solnik B, Moitra S, Kochanski G, Karro J, Sculley D (2017) Google vizier: a service for black-box optimization. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1487–1495

Guo J, Yang D, Siegmund N, Apel S, Sarkar A, Valov P, Czarnecki K, Wasowski A, Yu H (2018) Data-efficient performance learning for configurable systems. Empirical Softw Eng (EMSE) 23(3):1826–1867CrossRef

Herodotou H, Lim H, Luo G, Borisov N, Dong L, Cetin FB, Babu S (2011) Starfish: a self-tuning system for big data analytics. In: Conference on innovative data systems research

Hess MR, Kromrey JD (2004) Robust confidence intervals for effect sizes: a comparative study of cohen’sd and cliff’s delta under non-normality and heterogeneous variances. In: Annual meeting of the American educational research association, pp 1–30

Holland JH (1992) Genetic algorithms. Sci Amer 267(1):66–73CrossRef

Huband S, Hingston P, While L, Barone L (2003) An evolution strategy with probabilistic mutation for multi-objective optimisation. In: The 2003 congress on evolutionary computation, 2003. CEC’03., IEEE, vol 4, pp 2284–2291

Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: 5th LION

Jiarpakdee J, Tantithamthavorn C, Grundy J (2021) Practitioners’ perceptions of the goals and visual explanations of defect prediction models. arXiv:210212007

Kaltenecker C, Grebhahn A, Siegmund N, Apel S (2020) The interplay of sampling and machine learning for software performance prediction. IEEE Softw 37(4):58–66CrossRef

Kendall MG (1948) Rank correlation methods

Kolesnikov S, Siegmund N, Kästner C, Grebhahn A, Apel S (2019) Tradeoffs in modeling performance of highly configurable software systems. Softw Syst Model 18(3):2265–2283CrossRef

Laumanns M, Thiele L, Deb K, Zitzler E (2002) Combining convergence and diversity in evolutionary multiobjective optimization. Evolution Computat

Macbeth G, Razumiejczyk E, Ledesma RD (2011) Cliff’s delta calculator: a non-parametric effect size program for two groups of observations. Univ Psychol 10(2):545–555CrossRef

Mittas N, Angelis L (2012) Ranking and clustering software cost estimation models through a multiple comparisons algorithm. IEEE Trans Softw Eng 39(4):537–551CrossRef

Nair V, Menzies T, Siegmund N, Apel S (2017) Using bad learners to find good configurations. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp 257–267

Nair V, Yu Z, Menzies T, Siegmund N, Apel S (2018) Finding faster configurations using flash. IEEE Trans Softw Eng 46(7):794–811CrossRef

Sarkar A, Guo J, Siegmund N, Apel S, Czarnecki K (2015) Cost-efficient sampling for performance prediction of configurable systems (t). In: 2015 30th IEEE/ACM international conference on automated software engineering (ASE). IEEE, pp 342–352

Sawyer R (2011) Bi’s impact on analyses and decision making depends on the development of less complex applications. IJBIR 2:52–63. https://doi.org/10.4018/IJBIR.2011070104

Shrikanth N, Menzies T (2020) Assessing practitioner beliefs about software defect prediction. In: 2020 IEEE/ACM 42nd international conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 182–190

Siegmund N, Grebhahn A, Apel S, Kästner C (2015) Performance-influence models for highly configurable systems. In: Proceedings of the joint meeting on foundations of software engineering (ESEC/FSE), ACM, pp 284–294

Snoek J, Larochelle H, Adams R (2012) Practical bayesian optimization of machine learning algorithms. In: NIPS - volume 2

Song W, Chan FT (2015) Multi-objective configuration optimization for product-extension service. J Manuf Syst 37:113–125CrossRef

Tan SY, Chan T (2016) Defining and conceptualizing actionable insight: a conceptual framework for decision-centric analytics. arXiv:160603510

Tu H, Papadimitriou G, Kiran M, Wang C, Mandal A, Deelman E, Menzies T (2021) Mining workflows for anomalous data transfers. In: 2021 IEEE/ACM 18th international conference on mining software repositories (MSR), pp 1–12. https://doi.org/10.1109/MSR52588.2021.00013

Van Aken D, Pavlo A, Gordon GJ, Zhang B (2017) Automatic database management system tuning through large-scale machine learning. In: International conference on management of data, ACM

Van Veldhuizen DA (1999) Multiobjective evolutionary algorithms: classifications, analyses and new innovations. Tech rep, Air force inst of tech wright-pattersonafb oh school of engineering

Xia T, Krishna R, Chen J, Mathew G, Shen X, Menzies T (2018) Hyperparameter optimization for effort estimation. arXiv:180500336

Xu T, Jin L, Fan X, Zhou Y, Pasupathy S, Talwadker R (2015) Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software. In: Foundations of software engineering

Zhang Q, Li H (2007) Moea/d: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evolution Computat 11(6):712–731CrossRef

Zhu H, Jin J, Tan C, Pan F, Zeng Y, Li H, Gai K (2017) Optimized cost per click in taobao display advertising. arXiv preprint

Zitzler E, Laumanns M, Thiele L (2001) Spea2: improving the strength pareto evolutionary algorithm. TIK-Report, vol 103

Zuluaga M, Krause A, Püschel M (2016) ε-pal: an active learning approach to the multi-objective optimization problem. J Mach Learn Res 17(1):3619–3650MathSciNetMATH

Titel: VEER: enhancing the interpretability of model-based optimizations
verfasst von: Kewen Peng
Christian Kaltenecker
Norbert Siegmund
Sven Apel
Tim Menzies
Publikationsdatum: 01.06.2023
Verlag: Springer US
Erschienen in: Empirical Software Engineering / Ausgabe 3/2023
Print ISSN: 1382-3256
Elektronische ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-023-10296-w

Springer Professional

Abstract

Context:

Objective:

Method:

Results:

Conclusion:

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Weitere Artikel der Ausgabe 3/2023

Automatic identification of self-admitted technical debt from four different sources

Integrating human values in software development using a human values dashboard

Software testing in the machine learning era

Studying the characteristics of SQL-related development tasks: An empirical study

Automated test generation for Scratch programs

Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models

Premium Partner