Nicholas Gould
Abstract
In recent years, performance profiles have become a popular and widely used tool for benchmarking and evaluating the performance of several solvers when run on a large test set. Here we use data from a real application as well as a simple artificial example to illustrate that caution should be exercised when trying to interpret performance profiles to assess the relative performance of the solvers.
- E. D. Dolan and J. J. Moré. 2002. Benchmarking optimization software with performance profiles. Mathematical Programming 91, 2 (2002), 201--213.Google Scholar
Cross Ref
- N. I. M. Gould and J. A. Scott. 2015a. The State-of-the-Art of Preconditioners for Sparse Linear Least Squares Problems. Technical Report RAL-P-2015-10. Rutherford Appleton Laboratory.Google Scholar
- N. I. M. Gould and J. A. Scott. 2015b. The State-of-the-Art of Preconditioners for Sparse Linear Least Squares Problems: The Complete Results. Technical Report RAL-TR-2015-09. Rutherford Appleton Laboratory.Google Scholar
- N. J. Higham. 2009. The scaling and squaring method for the matrix exponential revisited. SIAM Review 51, 4 (2009), 747--767. Google ScholarDigital Library
- J. J. Moré and S. M. Wild. 2009. Benchmarking derivative-free optimization algorithms. SIAM Journal on Optimization 20, 1 (2009), 172--191. Google ScholarDigital Library
Index Terms
- A Note on Performance Profiles for Benchmarking Software
Recommendations
Green Fuzzer Benchmarking
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and AnalysisOver the last decade, fuzzing has been increasingly gaining traction due to its effectiveness in finding bugs. Nevertheless, fuzzer evaluations have been challenging during this time, mainly due to lack of standardized benchmarking. Aiming to ...
SPEC HPG benchmarks for high-performance systems
In this paper, we discuss the results and characteristics of the benchmark suites maintained by the Standard Performance Evaluation Corporation's (SPEC) High-Performance Group (HPG). Currently, SPECHPGhas two lines of benchmark suites for measuring ...
Benchmarking data warehouses
Database benchmarks can either help users in comparing the performances of different systems, or help engineers in testing the effect of various design choices. In the field of data warehouses, the Transaction Processing Performance Council's standard ...
Comments