A formal framework to analyze cost and performance in Map-Reduce based applications

Abstract

Satisfying the global throughput targets of scientific applications is an important challenge in high performance computing (HPC) systems. The main difficulty lies in the high number of parameters having an important impact on the overall system performance. These include the number of storage servers, features of communication links, and the number of CPU cores per node, among many others.

In this paper we present a model that computes a performance/cost ratio using different hardware configurations and focusing on scientific computing. The main goal of this approach is to balance the trade-off between cost and performance using different combinations of components for building the entire system. The main advantage of our approach is that we simulate different configurations in a complex simulation platform. Therefore, it is not necessary to make an investment until the system computes the different alternatives and the best solutions are suggested. In order to achieve this goal, both the system's architecture and Map-Reduce applications are modeled. The proposed model has been evaluated by building complex systems in a simulated environment using the SIMCAN simulation platform.

Introduction

During the last decade, most efforts in scientific applications were focused on supercomputers, grids, and commodity clusters [1], [2]. Generally, scientific applications are computation and data intensive, which means that these applications require long execution time. However, the execution time can be reduced by exploiting parallelism in CPU and I/O, especially, in cluster systems. Examples of those applications can be found in many fields like MRI scan data [3], astronomy [4], and Earthquake Modeling [5]. Due to its high cost effectiveness, commodity clusters have been the main resource for high performance computing (HPC). As an example, in the most recent survey of the fastest 500 computers in the world [6], 81.4% are clusters.

Scientific high-performance applications have reached a turning point where computing power is no longer the most important concern [7], [8]. Processing power has been doubling every 18 months for many years, enabling HPC applications to create, analyze and use increasingly large volumes of data. The impact of these trends on system architectures has caused a subsequent increase of the data-set sizes, which have to be stored and accessed efficiently. Therefore, programming applications that combine the exploitation of CPU parallelism with massive data operations is hard and a problem that has not yet resolved. Today, however, the emphasis is shifting from focusing the main optimization only on the computing system, to optimizing other systems such as the storage and the networking systems.

In late 2007, a new dimension was added to the scientific community: cloud computing. Although the concept of cloud computing emerged in 2007, there does not exist yet an accurate definition for it [1]. Cloud computing can be seen as a paradigm that provides access to a flexible and on-demand computing infrastructure, by allowing the user to rent a set of virtual machines to solve a computational problem.

In order to make a good use of resources, both clusters and cloud systems need a flexible and scalable method for developing and executing high performance scientific applications. Clearly, this common piece is the Map-Reduce scheme [9], a programming model for processing large data-sets in distributed systems. Although the Map-Reduce model was originally developed by Google, it is currently used by important companies like Amazon, Facebook, Baidu, Yahoo!, Last.fm, New York Times and AOL, becoming the de facto standard for large-scale data processing [10], [11]. From now on, the term scientific cluster is used to refer to internal data centers not made available to the general public, and public cloud refers to services that can be outsourced by paying each deployed virtual machine (VM) per unit of time basis.

This paper describes a model that splits the cluster configuration in four independent systems: CPU, storage, memory and network. Basically, this model takes as input a list of components for each system and produces, as output, a performance/cost ratio for each configuration. Accordingly, a set of detailed parameters related to the underlying characteristics (including cost) of each component must be specified. Thus, this model uses the information provided as input by the user to generate automatically a combination of system models to simulate its underlying hardware features. This model has been implemented in a simulated environment using the SIMCAN simulation platform [12], an open-source project for modeling and simulating distributed systems and applications. This platform also allows user to develop parallel and distributed applications, which is a powerful tool for calculating the performance of scientific computation using different hardware configurations [13], [14], [15], [16].

The results obtained from the application of our methodology are reported on this paper, supported by the SIMCAN simulation platform to optimize the balance and performance for scientific systems. The main contributions of this work can be summarized as:

• In contrast with existing work, mainly focusing on both optimizations and performance evaluation of Map-Reduce based applications [17], [18], our proposed model focuses on the relationship between the hardware cost, the configuration of the system, the overall application performance, and the user requirements. Accordingly, our work presents the design and development of an accurate model for optimizing the trade-offs between cost and performance of scientific clusters based on the requirements defined by the user.

• In order to show the accuracy of the proposed model, a big set of validation experiments has been exercised in real environments.

• A complete performance evaluation model allows to compare different hardware configurations. Thus, this model computes the best choice of the possible hardware alternatives for the specified requirements.

This paper represents an extended and enhanced version of previous work on the optimization of the trade-offs between cost and performance in cluster systems [19]. We have included a detailed description of the simulation platform used to perform the experiments described in this paper. We have extended the evaluation process that was included in our previous work with new experiments. Additionally, several charts that show the performance of the simulated scenarios have been included. These charts are very helpful to provide relevant information to the reader about the cost of obtaining the expected results.

The rest of the paper is structured as follows. Section 2 describes the motivation for this work. Section 3 presents an overview of SIMCAN. Section 4 describes a model to optimize the trade-offs between cost and performance in cluster systems, which is divided in three different parts: modeling of the system architecture, modeling of the Map-Reduce application, and the architecture of the complete model. Section 5 shows both validation and performance experiments using the proposed model. Section 6 presents related work. Finally, Section 7 presents our conclusions and some directions for future work.