Kenneth Hoste
Koen De Bosschere
ABSTRACT
A key challenge in benchmarking is to predict the performance of an application of interest on a number of platforms in order to determine which platform yields the best performance. This paper proposes an approach for doing this. We measure a number of microarchitecture-independent characteristics from the application of interest, and relate these characteristics to the characteristics of the programs from a previously profiled benchmark suite. Based on the similarity of the application of interest with programs in the benchmark suite, we make a performance prediction of the application of interest. We propose and evaluate three approaches (normalization, principal components analysis and genetic algorithm) to transform the raw data set of microarchitecture-independent characteristics into a benchmark space in which the relative distance is a measure for the relative performance differences. We evaluate our approach using all of the SPEC CPU2000 benchmarks and real hardware performance numbers from the SPEC website. Our framework estimates per-benchmark machine ranks with a 0.89 average and a 0.80 worst case rank correlation coefficient.
- M. Annavaram, R. Rakvic, M. Polito, J. Bouguet, R. Hankins, and B. Davies. The fuzzy correlation between code and performance predictability. In Proceedings of the 37th International Symposium on Microarchitecture (MICRO), pages 93--104, Dec. 2004. Google Scholar
Digital Library
- I. K. Chen, J. T. Coffey, and T. N. Mudge. Analysis of branch prediction via data compression. In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS VII), pages 128--137, Oct. 1996. Google ScholarDigital Library
- T. Conte. Insight, not (random) numbers. Keynote talk at the 2005 International Symposium on Performance Analysis of Systems and Software (ISPASS), Mar. 2005.Google Scholar
- L. Eeckhout, J. Sampson, and B. Calder. Exploiting program microarchitecture independent characteristics and phase behavior for reduced benchmark suite simulation. In Proceedings of the 2005 IEEE International Symposium on Workload Characterization (IISWC), pages 2--12, Oct. 2005.Google ScholarCross Ref
- L. Eeckhout, H. Vandierendonck, and K. De Bosschere. Quantifying the impact of input data sets on program behavior and its applications. Journal of Instruction-Level Parallelism, 5, Feb. 2003. http://www.jilp.org/vol5.Google Scholar
- M. Franklin and G. S. Sohi. Register traffic analysis for streamlining inter-operation communication in fine-grain parallel processors. In Proceedings of the 22nd Annual International Symposium on Microarchitecture (MICRO-22), pages 236--245, Dec. 1992. Google ScholarDigital Library
- R. A. Johnson and D. W.Wichern. Applied Multivariate Statistical Analysis. Prentice Hall, Fifth edition, 2002. Google ScholarDigital Library
- T. S. Karkhanis and J. E. Smith. A first-order superscalar processor model. In Proceedings of the 31st Annual International Symposium on Computer Architecture (ISCA-31), pages 338--349, June 2004. Google ScholarDigital Library
- J. Lau, J. Sampson, E. Perelman, G. Hamerly, and B. Calder. The strong correlation between code signatures and performance. In Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS), Mar. 2005. Google ScholarDigital Library
- J. Lau, S. Schoenmackers, and B. Calder. Structures for phase classification. In Proceedings of the 2004 International Symposium on Performance Analysis of Systems and Software (ISPASS), Mar. 2004. Google ScholarDigital Library
- A. Phansalkar and L. K. John. Performance prediction using program similarity. In Proceedings of the 2006 SPEC Benchmark Workshop, Jan. 2006.Google Scholar
- A. Phansalkar, A. Joshi, L. Eeckhout, and L. K. John. Measuring program similarity: Experiments with SPEC CPU benchmark suites. In Proceedings of the 2005 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS'05), pages 10--20, Mar. 2005. Google ScholarDigital Library
- R. H. Saavedra and A. J. Smith. Analysis of benchmark characteristics and benchmark performance prediction. ACM Transactions on Computer Systems, 14(4):344--384, Nov. 1996. Google ScholarDigital Library
- T. Sherwood, E. Perelman, G. Hamerly, and B. Calder. Automatically characterizing large scale program behavior. In Proceedings of the Tenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-X), pages 45--57, Oct. 2002. Google ScholarDigital Library
- A. Srivastava and A. Eustace. ATOM: A system for building customized program analysis tools. Technical Report 94/2, Western Research Lab, Compaq, Mar. 1994.Google ScholarDigital Library
- H. Vandierendonck and K. De Bosschere. Many benchmarks stress the same bottlenecks. In Proceedings of the Seventh Workshop on Computer Architecture Evaluation using Commercial Workloads (CAECW), pages 57--64, Feb. 2004.Google Scholar
- J. J. Yi, D. J. Lilja, and D. M. Hawkins. A statistically rigorous approach for improving simulation methodology. In Proceedings of the Ninth International Symposium on High Performance Computer Architecture (HPCA-9), pages 281--291, Feb. 2003. Google ScholarDigital Library
- Y. Zhong, S. G. Dropsho, and C. Ding. Miss rate prediction across all program inputs. In Proceedings of the Twelveth International Conference on Parallel Architectures and Compilation Techniques (PACT), Sept. 2003. Google ScholarDigital Library
Index Terms
- Performance prediction based on inherent program similarity
Recommendations
Measuring Benchmark Similarity Using Inherent Program Characteristics
This paper proposes a methodology for measuring the similarity between programs based on their inherent microarchitecture-independent characteristics, and demonstrates two applications for it: 1) finding a representative subset of programs from ...
Requirements-driven database systems benchmark method
Benchmarks are the vital tools in the performance measurement, evaluation, and comparison of relational database management systems (RDBMS). Standard benchmarks such as the TP1, TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, Wisconsin, and AS3 Ap ...
A Benchmark Characterization of the EEMBC Benchmark Suite
Benchmark consumers expect benchmark suites to be complete, accurate, and consistent, and benchmark scores serve as relative measures of performance. However, it is important to understand how benchmarks stress the processors that they aim to test. This ...
Comments