Carsten Binnig
ABSTRACT
Traditionally, the goal of benchmarking a software system is to evaluate its performance under a particular workload for a fixed configuration. The most prominent examples for evaluating transactional database systems as well as other components on top (such as a application-servers or web-servers) are the various TPC benchmarks.
In this paper we argue that traditional benchmarks (like the TPC benchmarks) are not sufficient for analyzing the novel cloud services. Moreover, we present some initial ideas how such a new benchmark should look like that fits better to the characteristics of cloud computing (e.g., scalability, pay-per-use and fault-tolerance). The main challenge of such a new benchmark is to make the reported results comparable because different providers offer different services with different capabilities and guarantees.
- 28msec, Inc. Sausalito, Apr. 2009. http://www.28msec.com/.Google Scholar
- Apache. HBase, Apr. 2009. http://hadoop.apache.org/hbase/.Google Scholar
- M. Armbrust et al. Above the clouds: A berkeley view of cloud computing. Technical Report UCB/EECS-2009-28, 2009.Google Scholar
- M. Brantner, D. Florescu, D. A. Graf, D. Kossmann, and T. Kraska. Building a database on S3. In Proc. of SIGMOD, 2008. Google ScholarDigital Library
- E. A. Brewer. Towards robust distributed systems. In Proc. of PODC, page 7, 2000. Google ScholarDigital Library
- M. Cafarella et al. Data management projects at Google. ACM SIGMOD Record, 37(1):34--38, 2008. Google ScholarDigital Library
- J.-D. Cryans, A. April, and A. Abran. Criteria to Compare Cloud Computing with Current Database Technology. In IWSM/Metrikon/Mensura '08: Proceedings of the International Conferences on Software Process and Product Measurement, pages 114--126, 2008. Google ScholarDigital Library
- Facebook. Cassandra - A distributed structured storage system, Apr. 2009. http://cwiki.apache.org/confluence/display/CSDR/Index.Google Scholar
- S. Garfinkel. An evaluation of Amazon's grid computing services: EC2, S3 and SQS, 2007.Google Scholar
- S. Gilbert and N. Lynch. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News, 33(2):51--59, 2002. Google ScholarDigital Library
- B. Hayes. Cloud computing. Commun. ACM, 51(7):9--11, 2008. Google ScholarDigital Library
- Heroku. Heroku - Instant Ruby Platform, Apr. 2009. http://heroku.com/.Google Scholar
- TPC. TPC-W Benchmark 1.8. http://www.tpc.org/tpcw/, 2002.Google Scholar
- TPC. TPC-H Benchmark 2.8. http://www.tpc.org/tpch/, 2008.Google Scholar
- TPC. TPC-C Benchmark 5.10.1. http://www.tpc.org/tpch/, 2009.Google Scholar
- W. Vogels. Data access patterns in the Amazon.com technology platform. In Proc. of VLDB, page 1, Sep 2007. Google ScholarDigital Library
Index Terms
- How is the weather tomorrow?: towards a benchmark for the cloud
Recommendations
HPC in Weather Forecast: Moving to the Cloud
Many traditional High-Performance Computing systems rely on hardware performance as well as compiler parallelization. The increased demand on computing resources rapidly escalates costs, which are the bigger limitation nowadays in both public and ...
Mark-copy: fast copying GC with less space overhead
OOPSLA '03: Proceedings of the 18th annual ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applicationsCopying garbage collectors have a number of advantages over non-copying collectors, including cheap allocation and avoiding fragmentation. However, in order to provide completeness (the guarantee to reclaim each garbage object eventually), standard ...
Mark-copy: fast copying GC with less space overhead
Special Issue: Proceedings of the OOPSLA '03 conferenceCopying garbage collectors have a number of advantages over non-copying collectors, including cheap allocation and avoiding fragmentation. However, in order to provide completeness (the guarantee to reclaim each garbage object eventually), standard ...
Comments