A framework for building hypercubes using MapReduce

Abstract

The European Space Agency’s Gaia mission will create the largest and most precise three dimensional chart of our galaxy (the Milky Way), by providing unprecedented position, parallax, proper motion, and radial velocity measurements for about one billion stars. The resulting catalog will be made available to the scientific community and will be analyzed in many different ways, including the production of a variety of statistics. The latter will often entail the generation of multidimensional histograms and hypercubes as part of the precomputed statistics for each data release, or for scientific analysis involving either the final data products or the raw data coming from the satellite instruments.

In this paper we present and analyze a generic framework that allows the hypercube generation to be easily done within a MapReduce infrastructure, providing all the advantages of the new Big Data analysis paradigm but without dealing with any specific interface to the lower level distributed system implementation (Hadoop). Furthermore, we show how executing the framework for different data storage model configurations (i.e. row or column oriented) and compression techniques can considerably improve the response time of this type of workload for the currently available simulated data of the mission.

In addition, we put forward the advantages and shortcomings of the deployment of the framework on a public cloud provider, benchmark against other popular solutions available (that are not always the best for such ad-hoc applications), and describe some user experiences with the framework, which was employed for a number of dedicated astronomical data analysis techniques workshops.

Introduction

Computer processing capabilities have been growing at a fast pace following Moore’s law, i.e. roughly doubling every two years during the last decades. Furthermore, the amount of data managed has been also growing at the same time as disk storage becomes cheaper. Companies like Google, Facebook, Twitter, LinkedIn, etc. nowadays deal with larger and larger data sets which need to be queried on-line by users and also have to answer business related questions for the decision making process. As instrumentation and sensors are basically made of the same technology as computing hardware, this has happened as well in science as we can discern in projects like the human genome, meteorology information and also in astronomical missions and telescopes like Gaia [1], Euclid  [2], the Large Synoptic Survey Telescope–LSST  [3] or the Square Kilometer Array–SKA  [4], which will produce data sets ranging from a petabyte for the entire mission in the case of Gaia to 10 petabytes of reduced data per day in the SKA.

Furthermore, raw data (re-)analysis is becoming an asset for scientific research as it opens up new possibilities to scientists that may lead to more accurate results, enlarging the scientific return of every mission. In order to cope with the large amount of data, the approach to take has to be different from the traditional one in which the data is requested and afterwards analyzed (even remotely). One option is to move to Cloud environments where one can upload the data analysis work flows so that they run in a low-latency environment and can access every single bit of information.

Quite a lot of research has been going on to address these challenges and new computing paradigms have lately appeared such as NoSQL databases, that relax transaction constraints, or other Massively Parallel Processing (MPP) techniques such as MapReduce  [5]. This new architecture emphasizes the scalability and availability of the system over the structure of the information and the savings in storage hardware this may produce. In this way the scale-up of problems is kept reasonably close to the theoretical linear case, allowing us to tackle more complex problems by investing more money in hardware instead of making new software developments which are always far more expensive. An interesting feature of this new type of data management system (MapReduce) is that it does not impose a declarative language (i.e. SQL), but it allows users to plug in their algorithms no matter the programming language they are written in and let them run and visit every single record of the data set (always brute force in MapReduce, although this may be worked around if needed by grouping the input data in different paths using certain constraints). This may also be accomplished to some extent in traditional SQL databases through User Defined Functions (UDFs) although code porting is always an issue as it depends a lot on the peculiarities of the database and debugging is not straightforward [6]. However, scientists and many application developers are more experienced at, or may feel more comfortable with, embedding their algorithms in a piece of software (i.e. a framework) that sits on top of the distributed system, while not caring much about what is going on behind the scenes or about the details of the underlying system.

Furthermore, some of the more widely used tools in data mining and statistics are multidimensional hypercubes and histograms, as they can provide summaries of different and complex phenomena (at a coarser or finer granularity) through a graphical representation of the data being analyzed, no matter how large the data set is. These tools are useful for a wide range of disciplines, in particular in science and astronomy, as they allow the study of certain features and their variations depending on other factors, as well as for data classification aggregations, pivot tables confronting two dimensions, etc. They also help scientists validate the generated data sets and check whether they fit within the expected values of the model or the other way around (also applicable to simulations).

As multidimensional histograms can be considered a very simple hypercube which normally contains one, two or three dimensions (often for visualization purposes) and whose measure is the count of objects given certain concrete values (or ranges) of its dimensions, we will generally refer to hypercubes through the paper and will only mention histograms when the above conditions apply (hypercubes with one to three dimensions whose only measure is the object count).

Previous work applying MapReduce to scientific data includes [7], where a High Energy Physics data analysis framework is embedded into the MapReduce system by means of wrappers (in the Map and Reduce phases) and external storage. The wrappers ensure that the analytical algorithms (implemented in a different programming language) can natively read the data in the framework specific format by copying it to the local file system or to other content distribution infrastructures outside the MapReduce platform. Furthermore,  [8] and [9] examine some of the current public Cloud computing infrastructures for MapReduce and study the effects and limitations of parallel applications porting to the Cloud respectively, both from a scientific data analysis perspective. In addition,  [10] shows that novel storage techniques being currently used in commercial parallel DBMS (i.e. column-oriented) can also be applied to MapReduce work flows, producing significant improvements in the response time of the data processing as well as in the compression ratios achieved for randomly-generated data sets. Last but not least, several general-purpose layers on top of Hadoop (i.e. Pig  [11] and Hive [12]) have lately appeared, aiming at processing and querying large data sets without dealing directly with the lower level API of Hadoop, but using a declarative language that gets translated into MapReduce jobs.

This paper is structured as follows. In Section  2, we present the simulated data set that will be used through the paper and some simple but useful examples that can be built with the framework. Section  3 describes the framework internals. In Section  4, we show the experiments carried out, analyzing the deployment in a public Cloud provider, examining the data storage models (including the column-oriented approach) and compression techniques, and benchmarking against two other well known approaches. Section  5 puts forward some user experiences in some astronomical data analysis techniques workshops. Finally, Sections  6 Conclusions, 7 Future work refer to the conclusions and future work respectively.