Performance Evaluation: Metrics, Models and Benchmarks

Server farms are ubiquitous in applications ranging from Web server farms to high-performance supercomputing systems to call centers. The popularity of the server farm architecture is understandable, as it allows for increased performance, while being cost-effective and easily scalable.

Given the prevalence of server farms, it is surprising that even at this late date so little is understood regarding their performance as compared with their single-server counterpart, particularly with respect to scheduling. Part of the problem is that there are at least three disjoint communities studying scheduling in server farms, including the SIGMETRICS community, the INFORMS community, and the SPAA/STOC/FOCS community, all of which have different approaches and goals. One of our goals in this talk is to make researchers aware of results in these different communities.

SAP is the world’s leading provider of business software. It delivers a comprehensive range of software products and services to its customers: Companies from all types of industries, ranging from small businesses to large, multinational enterprises engaged in global markets. The hardware and software requirements of these businesses are as diverse as the companies themselves, but for most customers they boil down to two key performance indicators: Throughput, of importance mainly for background processing, for example overnight payroll calculations, and response time, of relevance to end users actively engaged on the system. For over 15 years SAP and its hardware and technology partners have developed and used benchmarks to test the performance and scalability of both SAP solutions and the hardware they run on. The SAP Standard Application Benchmarks, first certified as such in 1995, help SAP and its partners prove that their software and hardware components scale up (and down) with their customers’ business needs, and support customers in configuring SAP Business Solutions for their productive systems.

Performance engineering of software could benefit from a closer integration of the use of performance models, and the use of measured data. Models can contribute to early warning of problems, exploration of solutions, and scalability evaluation, and when they are fitted to data they can summarize the data as a special powerful form of fitted function. Present industrial practice virtually ignores models, because of the effort to create them, and concern about how well they fit the system when it is implemented. The first concern is being met by automated generation from software specifications. The second concern can be met by fitting the models to data as it becomes available. This will adapt the model to the new situation and validate it, in a single step. The present paper summarizes the fitting process, using standard tools of nonlinear regression analysis, and shows it in action on examples of queueing and extended queueing models. The examples are a background for a discussion about the relationship between the models, and measurement data.

Recent developments in the analysis of stochastic process algebra models allow for transient measures of very large models to be extracted. By performing so-called fluid analysis of stochastic process algebra models, it is now feasible to analyse systems of size 10¹⁰⁰⁰ states and beyond. This paper seeks to extend the type of measure that can be extracted from this style of fluid analysis. We present a systematic transformation of a PEPA model that will allow us to extract measures analogous to response times. We end by extracting these response-time measures from a PEPA model of a healthcare system.

In this paper we explore the trade-off between security and performance in considering a model of a key distribution centre. The model is specified using the Markovian process algebra PEPA. The basic model suffers from the commonly encountered state space explosion problem, and so we apply some model reduction techniques and approximation to give a form of the model which is more scalable. The system is analysed numerically and results derived from the approximation are compared with simulation.

For component-based performance engineering, software component developers individually create performance specifications of their components. Software architects compose these specifications to architectural models. This enables assessing the possible fulfilment of performance requirements without the need to purchase and deploy the component implementations. Many existing performance models do not support component-based performance engineering but offer efficient solvers. On the other hand, component-based performance engineering approaches often lack tool support. We present a model transformation combining the advanced component concepts of the Palladio Component Model (PCM) with the efficient performance solvers of Layered Queueing Networks (LQN). Joining the tool-set for PCM specifications with the tool-set for LQN solution is an important step to carry component-based performance engineering into industrial practice. We validate the correctness of the transformation by mapping the PCM model of a component-based architecture to an LQN and conduct performance predictions.

Early, model-based performance predictions help to understand the consequences of design decisions on the performance of the resulting system before the system’s implementation becomes available. While this helps reducing the costs for redesigning systems not meeting their extra-functional requirements, performance prediction models have to abstract from the full complexity of modern hard- and software environments potentially leading to imprecise predictions. As a solution, the construction and execution of prototypes on the target execution environment gives early insights in the behaviour of the system under realistic conditions. In literature several approaches exist to generate prototypes from models which either generate code skeletons or require detailed models for the prototype. In this paper, we present an approach which aims at automated generation of a performance prototype based solely on a design model with performance annotations. For the concrete realisation, we used the Palladio Component Model (PCM), which is a component-based architecture modelling language supporting early performance analyses. For a typical three-tier business application, the resulting Java EE code shows how the prototype can be used to evaluate the influence of complex parts of the execution environment like memory interactions or the operating system’s scheduler.

This paper presents an approach and a corresponding tool for generating probabilistic and intensity-varying workload for Web-based software systems. The workload to be generated is specified in two types of models. An application model specifies the possible interactions with the Web-based software system, as well as all required low-level protocol details by means of a hierarchical finite state machine. Based on the application model, the probabilistic usage is specified in corresponding user behavior models by means of Markov chains. Our tool Markov4JMeter implements our approach to probabilistic workload generation by extending the popular workload generation tool JMeter. A case study demonstrates how probabilistic workload for a sample Web application can be modeled and executed using Markov4JMeter.

This work extends an existing workload comparison approach used for simulation based metrics to computer system based metrics. We apply this approach using processor hardware counters and compare characterizations of spec cpu2000 and spec cpu2006, against real (commercial) workloads and other benchmarks collected on the same computer system architecture. Using Independent Component analysis we reduce the many dimensional workload characterization space into a lesser dimensional representative space prior to comparing the distribution of workloads. We find that the spec cpu benchmarks are for the most part representative on the identified principal components, with notable exceptions.

An increasing number of synthetic topology generators are available, each claiming to produce representative Internet topologies. Every generator has its own parameters, allowing the user to generate topologies with different characteristics. However, there exist no clear guidelines on tuning the value of these parameters in order to obtain a topology with specific characteristics.

In this paper we optimize the parameters of several topology generators to match a given Internet topology. The optimization is performed either with respect to the link density, or to the spectrum of the normalized Laplacian matrix. Contrary to approaches in the literature that rely only on the largest eigenvalues, we take into account the set of all eigenvalues. However, we show that on their own the eigenvalues cannot be used to construct a metric for optimizing parameters. Instead we present a weighted spectral method which simultaneously takes into account all the properties of the graph.

Performance and scalability are essential characteristics of large-scale enterprise software. This paper presents the technologies behind the processes implemented at SAP. During the specification, design and implementation phases, Performance Design Patterns are used as guidelines, which also define the Key Performance Indicators (KPI) for performance and scalability tests. With proven scalability of software applications, SAP’s Sizing Process enables the transformation of business requirements into hardware requirements. It also allows SAP’s customers to flexibly configure their specific applications, on operating system (OS), database (DB), and hardware platforms of their choice. The SAP Standard Application Benchmarks are developed and executed to test the scalability in extremely high load situations and to verify the sizing statements from the sizing process. They are also used for SAP internal regression tests across releases, and by SAP’s hardware partners for platform tests. Besides the response time centric performance testing, analysis and optimization, SAP follows a KPI-focused approach which permits potential performance problems to be reliably predicted already in simple and easy-to-execute tests. The SAP NetWeaver Portal Benchmark is used to demonstrate how to conduct performance and scalability tests using single user tests and load tests. We will introduce the KPIs used for Java memory analysis and optimization. Finally, this paper shows how the results of these tests can be used in hardware sizing in customer implementation projects.

Web-Services based Service-Oriented Architectures (SOAs) become ever more important. The Web Services Reliable Messaging standard (WSRM) provides a reliable messaging layer to these systems. In this work we present parameters for acyclic continuous phase-type (ACPH) approximations for message transmission times in a WSRM implementation confronted with several different levels of IP packet loss. These parameters illustrate how large data sets may be represented by just a few parameters. The ACPH approximations presented here can be used for the stochastic modelling of SOA systems. We demonstrate application of the models using an M/PH/1 queue.

Web services are the building blocks of the emerging computing paradigm based on service-oriented architectures (SOAs). A web service is a self-describing, open component that supports rapid composition of distributed applications. In a SOA context, service providers are strategically interested both to predict and describe the QoS of the offered services. This paper introduces a model-driven approach to automatically predict and describe the QoS of composite web services specified by use of the Business Process Execution Language for Web Services. The paper is founded on a lightweight QoS-oriented extension of the WSDL and specifically addresses the QoS in terms of the performability attribute, which defines a combined measure of performance and reliability. The proposed approach is illustrated by use of an example application that shows how the performability analysis may lead to predictions that do not correspond to those obtained by approaches that only consider the performance attribute.

We consider a system of servers that process incoming requests. These requests experience periods of high and low arrival rate. Servers can be powered down dynamically to conserve power. We examine this system with a view to balancing the need between processing incoming requests quickly and reducing power consumption. The system is modeled formally and heuristics are presented to decide when servers should be powered down or up. Preliminary results of the performance of these heuristics are also included.

SPEC has recently released SPECpower_ssj2008, the first industry benchmark which measures performance and power of volume server class computers using graduated load levels. In this paper, we present a brief overview and an initial characterization of this benchmark by measuring the system resource utilization with the aid of processor monitoring events at graduated load levels and by comparing the sensitivity of final metric and other related data between various configurations consisting of hardware changes as well as software changes on Quad Core Intel Xeon processor based servers. Even though this is early data from a specific platform and OS, it still validates many expected patterns and opens exciting new opportunities for researchers to investigate specific areas as well as in-depth characterization as a next step.

Software response time distributions can be of high variance and multi-modal. Such characteristics reduce confidence or applicability in various statistical evaluations.

We contribute an approach to correlating response times to their corresponding operation execution sequence. This provides calling-context sensitive timing behavior models. The approach is based on three equivalence relations: caller-context, stack-context, and trace-context equivalence. To prevent model size explosion, a tree-based hierarchy provides timing behavior models that provide a trade-off between timing behavior model size and the amount of calling-context information considered.

In the case study, our approach provides response time distributions with significantly lower standard deviation, compared to using less or no calling-context information. An example from a performance analysis of an industry system demonstrates that multi-modal distributions can be replaced by multiple unimodal distributions using trace-context analysis.

A large Employee Appraisal System is accessed by clients from different international locations using the Internet. Being a distributed system, it is not easy to monitor and assess performance of such a large online transaction processing (OLTP) system. Performance Monitoring and Analysis of such a system requires pinpointing to the box or link in the system that is responsible for the overall slow performance. In this paper, we elaborate on our approach of performance monitoring and analysis of a large OLTP system using an employee appraisal system as an example.

In this work we augment the red-black tree implementation of STL set< ... >/map< ... > with a doubly linked list that is in sorted order. This is done for the purpose of speeding up C++ applications that use set< >/map< >::iterator considerably. In such cases, the doubly linked list helps in iterating over the set< >/map< > quickly. Usually the ++/– operations have an amortized cost of O(1) for a red-black tree implementation. The linked list augmentation helps in improving the ++/– operations to Θ(1). In addition, our experiments for IBM’s P5+ and P6 processors show that this mechanism improves performance for two SPEC CPU2006 benchmarks and there is no adverse cache effect when we support two additional pointers per node of a red-black tree.