Execution management in the GRID, for sensitivity studies of global climate simulations

Dynamical climate models are mathematical models that numerically solve the nonlinear equations governing the atmosphere on a global lattice with horizontal resolutions ranging from 50 to 300 km, depending on the application. These models require a set of initial conditions (values of climate variables—wind, pressure, temperature, etc,—on a lattice of points at the starting time) to propagate the solution forward in time.

In order to analyze the atmospheric part of the global climate system, we selected the CAM model (Community Atmosphere Model), which is the latest in a series of global atmosphere models developed at NCAR for the weather and climate research communities (Collins et al. 2004). CAM can be run for short (hours, days) or long periods of time (decades, centuries) to investigate the present, past or future climate variability. A great percentage of the total variability (70–90%) of global climate is obtained when an atmospheric model is forced with the oceanic fluxes; say the sea surface temperature and ice cover. As a first step, we ported the CAM model, which is an atmosphere-only model and requires lower boundary conditions at the surface (sea surface temperature). This model enables a wide range of experiments.

The port of a fully coupled model (such as the CCSM model which comprises CAM as the atmospheric part) is left as a natural future work.

The model can be run either in parallel (using MPI) or as a single process. The single-process version has been deployed and run in the EELA testbed with T42 resolution: 128 (longitude) × 64 (latitude) and 27 vertical levels, i.e. 221184 points per time step. The model produces 32 3D and 56 2D variables over the lattice. The simulation of a year takes approximately 48 hours of wall clock time in a 3 GHz Intel Pentium D processor, while 100 years would take around 7 months of wall clock time. The model produces 197 MB per time step, i.e. more than 720 GB per century. These figures increase when running the model at a more state-of-the-art resolution. T42 was state-of-the-art one decade ago and is only in use for very long simulations (centuries), but we used it for the development process.

The application we designed aims to perform sensitivity experiments by running an ensemble of CAM simulations with perturbed the sea surface temperatures as boundary conditions.

Climate models are complicated computer programs which require large amounts of CPU power. Most of them are parallelized. However, the GRID cannot make the most of this kind of parallelism, since the latency across geographically distributed computers would render the program completely inefficient. Apart from computer parallelism, climate science is recently making use of a large number of simulations, referred to as an “ensemble”, of the same phenomenon in order to assess the uncertainty inherent to the simulation (Hagedorn et al. 2005; Palmer 2002). Ensembles of simulations with varying parameters are also used for sensitivity experiments and many other applications. Each simulation in an ensemble is independent of the others and can be run asynchronously. This kind of parametric jobs is well suited for the GRID, since each simulation can be carried out in different nodes and the results are made available as a uniform data set in the Logical File Catalogue (LFC), ready to be analyzed.

Unlike volunteer computing projects, such as climateprediction.net (Allen 1999) where the GCM needs to be simplified and most of the results thrown away to avoid the overloading of the volunteer hosts, the GRID allows running a full state-of-the-art model and store the regular output information.

Nowadays, it is uncommon the use of GRID computing to run long-term jobs, due to the high rate of job failure and the CPU-time limitations for the jobs on the local management system (typically only jobs lasting less than 48 h are allowed). These problems become critical for long simulations such as those performed with climate models and other Earth Science applications. Thus, unlike many other applications ported to GRID, earth science applications need to make use of advanced techniques in workflow management. In particular, the climate application described in this paper has the following requirements:

The above requirements made necessary the development of a goal-oriented workflow manager in order to run the experiments with a minimum of human intervention. Therefore, we developed the GRID-CAM application which is a “GRID workflow management layer for CAM simulation”.

The requirements of the climate model described in this section are only an example of the needs of many other Earth Science applications.

In this section we briefly introduce and define the different components involved in a typical climate simulation on the GRID. We define an experiment as an ensemble of simulations (parametric jobs) designed to answer some scientific question (a single execution is the simplest experiment); each of the ensemble executions is called a realization and requires a set of input data to run the model in the prescribed simulation period (typically several years). A particular type of experiment is that related to climate sensitivity studies. In this case, the different sets of input data are obtained from a single one including certain user-defined perturbations to form the ensemble (perturbed initial or boundary conditions, perturbed parameters, perturbed radiative forcing, etc.).

Figure 1 shows a scheme illustrating the relationships between the main components of the application.

Therefore, climate simulation on the GRID requires the management of a complex workflow formed by experiments composed of realizations split across jobs. This workflow is not trivially managed by the currently available GRID middleware, so a new layer is necessary for a proper execution of climate simulations.

Climate models are mature applications with thousands of lines of code (usually Fortran). We introduced small modifications to the code to perform system calls to specific applications which are in charge of interacting with the GRID on behalf of the climate model. To this aim, we developed a new software layer, referred to as GRID Enabling Layer (GEL), which provides the model with the ability to interact with the GRID. The slightly modified source code of the model plus its GEL conform a fully featured GRID application. Since climate models are developed by external institutions, this approach is the best suited to keep up with the most recent updates with the least effort: only the small modifications to interact with the GEL need to be introduced at key points of any new release.

The GEL provides the following capabilities:

The above issues were solved by introducing Fortran system calls at 4 specific points of the CAM source code. These calls execute the GEL scripts which carry out the previously mentioned tasks. The slight modifications to the Fortran source code also allow us use fast-development programming languages for the GEL scripts (we used a mixture of shell, Perl and Python).

As we pointed out in “Requirements and workflow management”, there are several reasons which may break the flow of the job. In fact, during the first stages of the porting of the application, we did not account for all these possible errors. The result was that very few of our simulations finished.

To better understand the WML, we describe next the steps a GRID job goes through in our system since a user sends it to the scheduler until it is finished.

When a user submits a job from the UI, the scheduler sends it to a CE. Once there, the job takes the pending status while the CE finds a WN for the job in the local site. If everything goes fine, the job starts its execution in the WN and finishes with the SUC final state (see Fig. 2).

Unfortunately, there are several situations where the job does not reach this status. In order to create a robust workflow, we identified the failure points that were not managed by the gLite middleware and we designed a workflow able to overcome the following situations:

All these final status (SUC, SUSP, ERR, MID.ERR, PERF, WALL) are stored in the AMGA database for each job for further analysis.

After analyzing several job managers, we found that GridWay meta-scheduler was the one that best fulfilled our requirements, since GridWay detects job failures for all of the problems mentioned before, and it is able to reschedule the failed jobs to another cluster. Moreover, once the re-scheduled job starts to run in the WN, a component of the WML developed queries the AMGA database to find the latest restart files for this realization in order to continue the simulation started by the previous job. We have also adopted an additional monitoring feature provided by GridWay: while the job is running in the WN, a monitor script (running also in the WN) checks the status of the job. This monitor can copy the output and error files of our job to the UI with a given frequency. In this way, from the UI we can accurately determine any failure in each of our realizations. When an ensemble of simulations is sent to the GRID, each realization of the ensemble is converted to a GridWay job that is sent to the scheduler. When GridWay receives the jobs, it searches the WN better suited to our application needs and chooses the best among them. To do so, it uses a powerful scheduling policy that takes into account the user requested requirements (memory, CPU, etc.) and a heuristic scheduling based on the jobs sent in the past. For instance, if all jobs sent to a CE failed, GridWay will not try to send jobs to that site for a long period.

The AMGA database has two different tasks in the application. On one hand, it is used to store the information generated by the experiments executed in the GRID. On the other hand, it is used for monitoring purposes, storing all the status information about each of the jobs as metadata information. The tables and relationships used by GRID-CAM are shown in the Fig. 3.

Some of them are also relevant to the workflow, as described below:

In our current status, the application allows us to run a variety of scientific experiments using CAM. As a first example, we ran an experiment of 50 CAM realizations with boundary conditions (sea surface temperature) as observed during the period from January 1997 until March 1998 (15 months). This period includes the strongest El Niño event observed to date. We investigated the sensitivity of the precipitation over south-American countries to modifications of the sea surface temperature (SST) by means of an ensemble of CAM realizations.

To create the ensemble, we perturbed the observed SST by adding a given spatial pattern scaled by a random number. The spatial pattern used to perturb the SST is shown in Fig. 4. It is the SST anomaly of the last two strongest El Niño events with respect to the climatological mean. For the 1997 event, this anomaly reached a maximum value of 2.5 K. We normalized this anomalous pattern by dividing by this maximum value. Then, we selected random scale factors in the range from −2.5 (normal conditions, since the perturbation is opposite and of similar intensity to the anomaly generated by El Niño) to 2.5 (double anomaly than in the 1997 event). A zero value, mean a forcing similar to that observed during 1997 year.

The initial conditions for the experiment correspond to a previous CAM run which started from climatological conditions on 1st January 1990, but was forced by the observed SST for one decade. The atmospheric and soil conditions on the 1st January 1997 from this run were used as initial conditions for the perturbed SST experiment.

The 10-year simulation showed a slight but clear precipitation response to El Niño conditions. However, the perturbed runs from our Grid experiment did not clearly respond to higher than observed SST anomalies (Fig. 5a). Figure 5 represents the mean precipitation for the period October through January, when the most intense anomalies are registered. When the precipitation at a single sensitive point is analyzed against the scale of the perturbation applied (Fig. 5a), the response to increased El Niño conditions is clear until a value 0 of the perturbation scale (Niño’97 conditions) but seems to remain stable to stronger perturbations. These can also be seen in Fig. 5b and c where the whole South American distribution of precipitation is presented for a Niño’97-like realization and a stronger (double) anomaly realization. These are preliminary results which still need to be analyzed in a more systematic manner, using a larger number of simulations for statistical significance, and probably require the use of a regional model to generate more reasonably the precipitation response over mountainous areas.

We presented the successful port of an atmospheric Global Climate Model to the GRID by using existing middleware solutions plus newly developed tools (Grid Enabling Layer and Workflow Management Layer) to account for specific requirements posed by this application. In doing so, several weaknesses of current middleware were identified. We showed how current GRID technology is immature and not completely well suited for the Earth Science community. Specifically, for climate models, it has several weaknesses such as a high failure rate and time simulation limits, which require the creation of new middleware able to handle these problems.

In spite of these weaknesses, there are some fields in earth science, including recent trends in ensemble forecasting or any other involving independent simulations, which could greatly benefit from the GRID.

In our case, the achievements made thanks to the GRID-enabled application could not have been reached with the means usually at out hand. Unlike local clusters or supercomputers, optimized for MPI and OpenMP jobs, the GRID is perfectly suited for independent simulations. In our case, it would have been impossible accessing a cluster with the number of CPUs available in the GRID. An even more important topic is the storage capacity; climate models manage huge files that need a lot of storage capacity. Thus, even though many jobs failed and needed to be rescheduled, the benefits from accessing a GRID composed of a large number of processors and disk space outweigh the drawbacks. We completed around 50 simulated years in 1 week.

The next steps in our work are porting the regional WRF model to the GRID and create a new application combining CAM and WRF. The idea is that the users specify a region to study and then run a cascade of CAM and WRF simulations for this region. In order to do this, improvements to the workflow will be necessary to manage the dependencies between the CAM and WRF jobs.

As a final remark, although the effort of developing an application in GRID is very high, the results are worth. Once the initial effort is done, many other applications with similar requirements can be much easily ported. This successful experience is promising and we will continue working in this line.