Abstract
Linux and other open-source Unix variants (and their distributors) provide researchers with full-fledged operating systems that are widely used. However, due to their complexity and rapid development, care should be exercised when using these operating systems for performance experiments, especially in systems research. In particular, the size and continual evolution of the Linux code-base makes it difficult to understand, and as a result, decipher and explain the reasons for performance improvements. In addition, the rapid kernel development cycle means that experimental results can be viewed as out of date, or meaningless, very quickly. We demonstrate that this viewpoint is incorrect because kernel changes can and have introduced both bugs and performance degradations.
This paper describes some of our experiences using Linux and FreeBSD as platforms for conducting performance evaluations and some performance regressions we have found. Our results show, these performance regressions can be serious (e.g., repeating identical experiments results in large variability in results) and long lived despite having a large negative effect on performance (one problem was present for more than 3 years). Based on these experiences, we argue: it is sometimes reasonable to use an older kernel version, experimental results need careful analysis to explain why a performance effect occurs, and publishing papers validating prior research is essential.
- M. Baker. Independent labs to verify high-profile papers. Nature | News, August 2012.Google Scholar
- S. M. Blackburn, A. Diwan, M. Hauswirth, P. F. Sweeney, J. N. Amaral, V. Babka, W. Binder, T. Brecht, L. Bulej, L. Eeckhout, S. Fischmeister, D. Frampton, R. Garner, A. Georges, L. J. Hendren, M. Hind, A. L. Hosking, R. Jones, T. Kalibera, P. Moret, N. Nystrom, V. Pankratius, and P. Tuma. Evaluate collaboratory technical report #1: Can you trust your experimental results?, February 2012. http://evaluate.inf.usi.ch/technical-reports/1.Google Scholar
- B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATEC '04, Berkeley, CA, U.S.A., 2004. USENIX Association. Google ScholarDigital Library
- CentOS: The community enterprise operating system. http://www.centos.org.Google Scholar
- J. Corbet, G. Kroah-Hartman, and A. McPherson. Linux kernel development: How fast it is going, who is doing it, what they are doing, and who is sponsoring it, Mar. 2012. http://go.linuxfoundation.org/who-writes-linux-2012.Google Scholar
- Evaluate Collaboratory. Experimental evaluation of software and systems in computer science. http://evaluate.inf.usi.ch/.Google Scholar
- D. G. Feitelson. Experimental computer science: The need for a cultural change, December 2006. http://www.cs.huji.ac.il/ feit/papers/exp05.pdf.Google Scholar
- A. S. Harji. Performance Comparison of Uniprocessor and Multiprocessor Web Server Architectures. PhD thesis, University of Waterloo, 2010. http://hdl.handle.net/10012/5040.Google Scholar
- A. S. Harji, P. A. Buhr, and T. Brecht. Our troubles with Linux and why you should care. In Proceedings of the Second Asia-Pacific Workshop on Systems, APSys '11, pages 2:1-2:5, New York, NY, USA, July 2011. ACM. Google ScholarDigital Library
- A. S. Harji, P. A. Buhr, and T. Brecht. Comparing high-performance multi-core web-server architectures. In Proceedings of the 5th Annual International Systems and Storage Conference (SYSTOR 2012), pages 2:1--2:12, New York, NY, USA, June 2012. ACM. Google ScholarDigital Library
- M. Larabel. Five Years Of Linux Kernel Benchmarks: 2.6.12 Through 2.6.37. Phoronix Media, Nov. 2010. http://www.phoronix.com/scan.php?page=article-&item=linux 2612 2637&num=1.Google Scholar
- J. Lehrer. The truth wears off: Is there something wrong with the scientific method? The New Yorker, December 13, 2010.Google Scholar
- Linux kernel performance! http://kernel-perf.sourceforge.net.Google Scholar
- J. N. Matthews. The case for repeated research in operating systems. SIGOPS Operating Systems Review, 38:5--7, April 2004. Google ScholarDigital Library
- Netcraft, Jan. 2012. https://ssl.netcraft.com/sslsample-report/CMatch/oscnt all.Google Scholar
- OProfile, 2012. http://oprofile.sourceforge.net.Google Scholar
- D. Pariag, T. Brecht, A. Harji, P. Buhr, and A. Shukla. Comparing the performance of web server architectures. In Proc. of the 2nd ACM SIGOPS/EuroSys Conf. on Computer Systems, pages 231--243. ACM, Mar. 2007. Google ScholarDigital Library
- C. Small, N. Ghosh, H. Saleeb, M. Seltzer, and K. Smith. Does systems research measure up? Technical report, Harvard University, TR-16-97, 1997.Google Scholar
- J. Summers, T. Brecht, D. L. Eager, and B. Wong. Methodologies for generating HTTP streaming video workloads to evaluate web server performance. In 5th Annual International Systems and Storage Conference (SYSTOR), pages 13:1--13:12, 2012. Google ScholarDigital Library
- J. Summers, T. Brecht, D. L. Eager, and B. Wong. To chunk or not to chunk: Implications for HTTP streaming video server performance. In 22nd SIGMM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), pages 15--20, Toronto, Canada, June 2012. Google ScholarDigital Library
- SUSE linux enterprise server for system z: The market leader for linux on ibm mainframes, May 2010.http://www.novell.com/docrep/2010/05/-SLES for system z market share leader.pdf.Google Scholar
- The SystemTap home page, 2010. http://sourceware.org/systemtap/.Google Scholar
- W. F. Tichy. Should computer scientists experiment more? Computer, 31(5):32--40, 1998. Google ScholarDigital Library
- Top 500 supercomputer sites. http://www.top500.org/statistics/list, Category→Operating System Family→Submit.Google Scholar
- Web Technology Surveys. Historical trends in the usage of Linux versions for websites, Aug. 2011. http://w3techs.com/technologies/history details/oslinux.Google Scholar
Recommendations
Our troubles with Linux and why you should care
APSys '11: Proceedings of the Second Asia-Pacific Workshop on SystemsLinux provides researchers with a full-fledged operating system that is widely used and open source. However, due to its complexity and rapid development, care should be exercised when using Linux for performance experiments, especially in systems ...
Comments