Joseph Yossi Gil
ABSTRACT
The extra abstraction layer posed by the virtual machine, the JIT compilation cycles and the asynchronous garbage collection are the main reasons that make the benchmarking of Java code a delicate task. The primary weapon in battling these is replication: "billions and billions of runs", is phrase sometimes used by practitioners. This paper describes a case study, which consumed hundreds of hours of CPU time, and tries to characterize the inconsistencies in the results we encountered.
- K. Arnold and J. Gosling. The Java Programming Language. Addison-Wesley, Reading, Massachusetts, 1996. Google Scholar
Digital Library
- M. Arnold, M. Hind, and B. G. Ryder. Online feedback-directed optimization of java. In Proc. of the 17th Ann. Conf. on OO Prog. Sys., Lang., & Appl. (OOPSLA'02). Google ScholarDigital Library
- H. Berry, D. Gracia Pérez, and O. Temam. Chaos in computer performance. Chaos, 16(1):013110--013110-15, 2006.Google ScholarCross Ref
- S. M. Blackburn, R. Garner, C. Hoffman, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In P. L. Tarr and W. R. Cook, eds., Proc. of the 21st Ann. Conf. on OO Prog. Sys., Lang., & Appl. (OOPSLA'06). Google ScholarDigital Library
- S. M. Blackburn, K. S. McKinley, R. Garner, C. Hoffmann, A. M. Khan, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanovik, T. VanDrunen, D. von Dincklage, and B. Wiedermann. Wake up and smell the coffee: evaluation methodology for the 21st century. Communications of the ACM, 51(8):83--89, 2008. Google ScholarDigital Library
- L. Eeckhout, A. Georges, and K. De Bosschere. How java programs interact with virtual machines at the microarchitectural level. In R. Crocker and G. L. S. Jr., eds., Proc. of the 18th Ann. Conf. on OO Prog. Sys., Lang., & Appl. (OOPSLA'03). Google ScholarDigital Library
- A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous Java performance evaluation. In R. P. Gabriel and D. Bacon, eds., Proc. of the 22nd Ann. Conf. on OO Prog. Sys., Lang., & Appl. (OOPSLA'07). Google ScholarDigital Library
- A. Georges, D. Buytaert, L. Eeckhout, and K. De Bosschere. Method-level phase behavior in Java workloads. In Vlissides and Schmidt {13}, pp. 270--287. Google ScholarDigital Library
- S. Hazelhurst. Truth in advertising : reporting performance of computer programs, algorithms and the impact of architecture and systems environment. South African Computer Journal, 46:24--37, 2010.Google ScholarCross Ref
- X. Huang, S. M. Blackburn, K. S. McKinley, J. E. B. Moss, Z. Wang, and P. Cheng. The garbage collection advantage: improving program locality. In Vlissides and Schmidt {13}, pp. 69--80. Google ScholarDigital Library
- D. Lea, D. F. Bacon, and D. Grove. Languages and performance engineering: method, instrumentation, and pedagogy. Sigplan Notices, 43:87--92, 2008. Google ScholarDigital Library
- T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Producing wrong data without doing anything obviously wrong! In Proceeding of the 14th international conference on Architectural support for programming languages and operating systems. Google ScholarDigital Library
- J. M. Vlissides and D. C. Schmidt, eds. Proc. of the 19th Ann. Conf. on OO Prog. Sys., Lang., & Appl. (OOPSLA'04), Vancouver, BC, Canada, Oct. 2004. ACM SIGPLAN Notices 39 (10).Google Scholar
Index Terms
- A microbenchmark case study and lessons learned
Recommendations
The DaCapo benchmarks: java benchmarking development and analysis
OOPSLA '06: Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applicationsSince benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection ...
SPEC MPI2007—an application benchmark suite for parallel systems using MPI
International Supercomputing Conference (ISC07)The SPEC High-Performance Group has developed the benchmark suite SPEC MPI2007 and its run rules over the last few years. The purpose of the SPEC MPI2007 benchmark and its run rules is to further the cause of fair and objective benchmarking of high-...
The DaCapo benchmarks: java benchmarking development and analysis
Proceedings of the 2006 OOPSLA ConferenceSince benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection ...
Comments