Building a National Distributed e-Infrastructure–PL-Grid

Frontmatter

PL-Grid: Foundations and Perspectives of National Computing Infrastructure

Abstract

The Polish Grid Initiative commenced in 2009 as part of the PL-Grid project funded within the framework of the Innovative Economy Operational Programme. The main objective is to provision a persistent heterogeneous computational platform for the Polish scientific community with a unified interface, enabling easy access to the distributed large-scale computing infrastructure. The project establishes a countrywide computing platform which supports scientific research through integration of experimental data and results of advanced computer simulations carried out by geographically distributed computer infrastructure and teams. The solutions applied in setting up this e-infrastructure facilitate integration with other, similar platforms around the world.

In this chapter the foundations of the PL-Grid architecture, a brief history of the project and its most important objectives are presented.

Jacek Kitowski, Michał Turała, Kazimierz Wiatr, Łukasz Dutka

Integrating Various Grid Middleware Components and User Services into a Single Platform

Abstract

The PL-Grid project is constructing a grid computing infrastructure supporting mature middleware suites such as gLite and UNICORE, as well as new ones like QosCosGrid. It also delivers various services, e.g. graphical tools and databases. Users need to be able to access all services and resources in a uniform way, if possible – with a single set of credentials. PL-Grid integrates the middleware and services accessible to users by delivering a coherent platform upon which to interact with the services. The required integration of procedures and tools is described in this chapter. The resulting platform provides the scientists conducting research with a wide variety of powerful tools and resources, enabling them to maximize their scientific potential.

Marcin Radecki, Tadeusz Szymocha, Daniel Harężlak, Maciej Pawlik, Jakub Andrzejewski, Wojciech Ziajka, Marcin Szelc

Towards Professional Service Operations in Grids

Abstract

Management of large distributed IT infrastructures requires a Service Operations framework covering day-to-day maintenance. In this chapter we describe the Service Operations model developed in the PL-Grid project. First, we sketch the processes that needed to be covered by Service Operations. Subsequently, we present the management solutions worked out in PL-Grid and describe how they were influenced by the Information Technology Infrastructure Library (ITIL). We also show how to adapt ITIL rules to manage geographically and administratively distributed infrastructures, namely those providing grid resources to the academic community. We claim that adapting a certain Service Operations model is a prerequisite of developing a mature infrastructure, ensuring user satisfaction.

Marcin Radecki, Tomasz Szepieniec, Tadeusz Szymocha, Magda Szopa, Małgorzata Krakowian

New Capabilities in QosCosGrid Middleware for Advanced Job Management, Advance Reservation and Co-allocation of Computing Resources – Quantum Chemistry Application Use Case

Abstract

In this chapter we present the new capabilities of QosCosGrid (QCG) middleware for advanced job and resource management in the grid environment. By connecting many computing clusters together, QosCosGrid offers easy-to-use mapping, execution and monitoring capabilities for a variety of complex computations, such as parameter sweep, workflows, MPI or hybrid MPI-OpenMP as well as multiscale simulations. Thanks to QosCosGrid, large-scale programming models written in Fortran, C, C++ or Java can be automatically distributed over a network of computing resources with guaranteed Quality of Service – for example guaranteed startup time of a job. Consequently, applications can be run at specified periods with reduced execution time and waiting times. This enables more complex problem instances to be addressed. In order to prove the usefulness of the new functionality of QosCosGrid a detailed description of the system along with a real use case scenario from the quantum chemistry science domain will be presented in this chapter.

Bartosz Bosak, Jacek Komasa, Piotr Kopta, Krzysztof Kurowski, Mariusz Mamoński, Tomasz Piontek

Seamless Access to the PL-Grid e-Infrastructure Using UNICORE Middleware

Abstract

This chapter provides a brief overview of the UNICORE grid middleware and its utilization in the large Distributed Computing Infrastructure. UNICORE framework, in its recent version implements key grid standards and specifications. The system architecture and capabilities, such as security, workflow and data management are described. The installation of the UNICORE environment in the PL-Grid is presented. Special attention is given to the integration of the UNICORE middleware with the PL-Grid authentication and authorization framework which allows for uniform infrastructure and user management across different middlewares. The solutions for monitoring and accounting of the UNICORE infrastructure is presented.

Krzysztof Benedyczak, Marcin Stolarek, Radosław Rowicki, Rafał Kluszczyński, Marcelina Borcz, Grzegorz Marczak, Maciej Filocha, Piotr Bała

User-Oriented Provisioning of Secure Virtualized Infrastructure

Abstract

The chapter presents a system for effective provisioning of VM Sets and supporting the dialog between providers and end users. VM Sets are networks of interconnected virtual appliances. Requirements for user-oriented provisioning of VM Sets are presented and an infrastructure supporting this activity is described. Stages of the VM Set provisioning process are defined and their functionality is described. Subsequently, manageability aspects are presented with special focus on runtime aspects. The implementation status of the VM Set provisioning environment within the PL-Grid Project is also reported.

Marcin Jarząb, Jacek Kosiński, Krzysztof Zieliński, Sławomir Zieliński

Flexible and Extendable Mechanism Enabling Secure Access to e-Infrastructures and Storage of Confidential Data for the GridSpace2 Virtual Laboratory

Abstract

This paper describes the security framework used by the GridSpace2 platform. The framework needs to restrict access to the Experiment Workbench to authorized users only as well as enable scientists to use multiple resources – computers (such as clusters, grids and clouds) and data, like e.g. relational database management systems (RDBMSes). The paper analyzes the appropriate technologies, details the proposed solution and summarizes the results of our research and development of flexible and extensible security solutions for scientists who need transparent access to heterogeneous compute and data resources. Additionally, as part of this paper, a wallet mechanism is described which enables secure storage of arbitrary confidential data such as credentials for external services.

Jan Meizner, Eryk Ciepiela, Piotr Nowakowski, Joanna Kocot, Maciej Malawski, Marian Bubak

SARA – System for Inventory and Static Security Control in a Grid Infrastructure

Abstract

IT security, even if once achieved, is not a permanent state but rather a process. One of the main factors that impact this process is the ability to identify security vulnerabilities in the software. Disclosing such a flaw is usually followed by issuing a patch. However, for maintainers of a heterogeneous and compound environment, being up to date with all necessary fixes, may be an unexpectedly difficult task. Developing custom software in a grid project introduces another dimension to this problem. The SARA system for static security control has been developed to help the administrators with that issue.

Gerard Frankowski, Michał Rzepka

ACARM-ng: Next Generation Correlation Framework

Abstract

ACARM-ng is an extensible, plug-in-based alert correlation framework. It introduces abstractions over correlation, reporting, reaction, gathering data from multiple sources and data storage. ACARM-ng supports real-time reporting, meaning that alerts can be reported while still being correlated. For an administrator, a Web User Interface is provided, to present gathered and correlated data in a consistent way. The system makes use of multi-core architectures and is written in C++.

Bartłomiej Balcerek, Bartosz Szurgot, Mariusz Uchroński, Wojciech Waga

Security Best Practices: Applying Defense-in-Depth Strategy to Protect the NGI_PL

Abstract

The role of security in modern IT systems is continuously growing. Large infrastructures have to be protected against sophisticated attacks on organizational, technical and logical levels. Achieving sufficient security level becomes even more difficult for distributed and, often, heterogeneous environments that involve valuable assets and data – like grids. The main goal of the work described within this paper is to provide maximum level of protection against network attackers to the PL-Grid – Polish National Grid Initiative – infrastructure.

Bartłomiej Balcerek, Gerard Frankowski, Agnieszka Kwiecień, Adam Smutnicki, Marcin Teodorczyk

Automation of System Monitoring Based on Fuzzy Logic or Rules; Comparison of Two Designed Approaches with Regard to Computational Infrastructures

Abstract

This paper is focused on monitoring automation of distributed systems. In the presented research, AI-based approaches to distributed monitoring related to large distributed systems such as grids, were explored. In both presented concepts knowledge is used to make decisions regarding management actions, using rules and fuzzy logic. The first concept is an agent-less rule-based solution, implemented in a high-level monitoring system called Saude-Net. It allows to define actions for monitored resources, using a kind of expert system. The second solution, which exploits agents and fuzzy logic, is realized in a system called SAMM Compliant Agent. Both presented systems are capable of reacting to observed failures and of modifying their knowledge to better fit possible problems with resources. We also present a short comparison of the two concepts, and an analysis of their usage.

Włodzimierz Funika, Filip Szura, Jacek Kitowski

A Toolkit for Storage QoS Provisioning for Data-Intensive Applications

Abstract

In this paper, we present a toolkit named FiVO/QStorMan for supporting data-intensive applications in distributed environments with storage-related Quality of Service provisioning. Main features of the toolkit are: explicit definition of non-functional requirements regarding storage resources, usage of semantic descriptions of the available storage infrastructure and active monitoring data concerning the infrastructure workload and users operations. In particular, we describe user interfaces of the toolkit along with descriptions of main parts of the toolkit. In addition, the paper describes results of the performed experimental evaluation of the toolkit which confirm the effectiveness of the proposed approach for the storage-related QoS provisioning.

Renata Słota, Dariusz Król, Kornel Skałkowski, Bartosz Kryza, Darin Nikołow, Michał Orzechowski, Jacek Kitowski

Implementation of Service Level Management in PL-Grid Infrastructure

Abstract

One of the key concept of Grids – as stated in their definition – is providing nontrivial quality of service. However, there are many users whose specific needs for guaranteed resources (depending on the type of their applications and research schedule) are not satisfied. This fact is the main motivation for introducing Service Level Management in the Polish National Grid Infrastructure (PL-Grid). The paper presents a model of SLAs with quality properties required in HPC Grid. In order to properly manage these SLAs a negotiation process has been proposed. Finally, an implementation of the procedures and tools needed to support this processes is summarized.

Tomasz Szepieniec, Małgorzata Tomanek, Marcin Radecki, Magda Szopa, Marian Bubak

Highly Integrated Environment for Parallel Application Development Using QosCosGrid Middleware

Abstract

QosCosGrid infrastructure has been prepared, introduced successfully on multiple clusters and comprehensively tested. In order to enhance user experience and increase usability, a set of additional tools must be prepared. Profiting from distributed software becomes more popular nowadays, but the number of tools supporting developers of parallel programs is still smaller than developers’ needs. Eclipse PTP is the leading open source tool among Integrated Development Environments for distributed software. In this chapter we shortly present Eclipse PTP and related Eclipse plug-ins aiding programmers in developing parallel software. Later we describe how QosCosGrid middleware has been integrated with Eclipse PTP. We present the plug-in basic concepts, such as the Resource Manager or Remote Services, general architecture, functionality of the PTP QCG plug-in and advantages over other solutions of this kind.

Bartosz Bosak, Jan Konczak, Krzysztof Kurowski, Mariusz Mamoński, Tomasz Piontek

User-Friendly Frameworks for Accessing Computational Resources

Abstract

The grid technology, even though it gives great computational power and greatly improves the manner of exploitation of resources, also has its disadvantages. Grid systems, due to their distribution and heterogeneity, are very complex and hard to access and oversee. Thus, one of the research fields designs tools and technologies that give easy, secure and consistent access to grid applications and resources as well as seamless interoperation between various computing environments. The lack of user-friendly tools is especially annoying in cases when users need to access computing resources of different grid infrastructures. Similar problems are experienced by grid application developers who should focus on applications themselves and not on their interoperation with different grid middleware. In this article we describe the concept of an abstract grid model and its implementation in two user-friendly frameworks – Migrating Desktop and g-Eclipse. They both are intuitive graphical environments that provide: easy access to heterogeneous resources and seamless interoperation of underlying middleware solutions. Although the two products provide similar functionalities they are complementary to each other and target different user groups. The method of integration of scientific applications with both frameworks was also presented.

Bartek Palak, Paweł Wolniewicz, Marcin Płóciennik, Michał Owsiak, Tomasz Żok

Online Web-Based Science Gateway for Nanotechnology Research

Abstract

The main objective of Science Gateways is to give users remote access to supercomputers and large-scale computing environments in an interactive, web-based and graphical manner. We present a tool, called Vine Toolkit, that has been successfully used as a core web platform for various Science Gateways. Vine Toolkit is a modular, extensible and easy-to-use tool as well as a high-level Application Programming Interface (API) for various applications, visualization components and building blocks. In a nutshell, it allows interoperability between many different HPC and grid technologies within the service layer. As a result, Vine Toolkit provides an ability to build a portal upon different HPC technologies working together to deliver a complete solution to the users. In this article, we briefly describe our most complex and feature-rich project – the Nanotechnology Gateway, as well as a set of tools relevant to advanced scientific portals, development of which was driven by various requirements defined by scientists and gathered in scope of the PL-Grid project.

Piotr Dziubecki, Piotr Grabowski, Michał Krysiński, Tomasz Kuczyński, Krzysztof Kurowski, Tomasz Piontek, Dawid Szejnfeld

Scripting Language Extensions Offered by the GridSpace Experiment Platform

Abstract

Many existing problem solving environments provide scientists with convenient methods for building scientific applications over distributed computational and storage resources. In many cases a basic set of features of such environments is sufficient to conduct a complete experiment flow. However, complex cases often require extensions supporting an external piece of software or a communication standard not integrated beforehand. Most environments deal with such cases by providing an extension facility and letting third parties add required features. The GridSpace environment also includes several mechanisms for extending its own functionality and here we describe how this can be accomplished. We focus on extensions already implemented such as local job submission and scripting language repositories, as well as on a GUI extension point which can be used to add custom graphical user interfaces to GridSpace experiments independently of their release process.

Daniel Harężlak, Marek Kasztelnik, Eryk Ciepiela, Marian Bubak

Managing Entire Lifecycles of e-Science Applications in the GridSpace2 Virtual Laboratory – From Motivation through Idea to Operable Web-Accessible Environment Built on Top of PL-Grid e-Infrastructure

Abstract

The GridSpace2 environment, developed in the scope of the PL-Grid Polish National Grid Initiative, constitutes a comprehensive platform which supports e-science applications throughout their entire lifecycle. Application development may involve multiple phases, including writing, prototyping, testing and composing the application. Once the application attains maturity it becomes operable and capable of being executed, although it may still be subject to further development – including actions such as sharing with collaborating researchers or making results publicly available with the use of dedicated publishing interfaces. This paper describes each of these phases in detail, showing how the GridSpace2 platform can assist the developers and publishers of computational experiments.

Eryk Ciepiela, Piotr Nowakowski, Joanna Kocot, Daniel Harężlak, Tomasz Gubała, Jan Meizner, Marek Kasztelnik, Tomasz Bartyński, Maciej Malawski, Marian Bubak

GridSpace2 Virtual Laboratory Case Study: Implementation of Algorithms for Quantitative Analysis of Grain Morphology in Self-assembled Hexagonal Lattices According to the Hillebrand Method

Abstract

This work presents the implementation of a method, originally proposed by Hillebrand et al. [1], of quantitative analysis of the grain morphology in self-assembled hexagonal lattices. This method can be effectively used for investigation of structural features as well as regular hexagonal arrangement of nanoporous alumina layers formed on the metal surface during the self-organized anodization process. The method has been implemented as a virtual experiment in the GridSpace2 Virtual Laboratory [15] which is a scientific computing platform developed in the scope of the PL-Grid [9] project. The experiment is a GridSpace2 pilot and therefore made available to the wider community of PL-Grid users. It is both editable and executable through a web portal offered by the GridSpace2 Experiment Workbench [17], dedicated to PL-Grid users. Moreover, since all GridSpace2 experiments are embeddable on arbitrary web sites owing to the Collage [16] feature, the final version of the experiment has been published as an executable publication [18] with execution rights granted to all PL-Grid users.

Eryk Ciepiela, Leszek Zaraska, Grzegorz D. Sulka

Examining Protein Folding Process Simulation and Searching for Common Structure Motifs in a Protein Family as Experiments in the GridSpace2 Virtual Laboratory

Abstract

This paper presents two in-silico experiments from the field of bioinformatics. The first experiment covers the popular problem of protein folding process simulation and investigates the correctness of the “Fuzzy Oil Drop” model (FOD) [3], on over 60 thousands of proteins deposited in Protein Data Bank [18]. The FOD model assumes the hydrophobicity distribution in proteins to be accordant with the 3D Gauss function differentiating the hydrophobicity density from the highest in the center of the molecule, to zero level on the surface. The second experiment focuses on performing comparison of proteins that belong to the same family. Examination of proteins alignment at three different levels of protein description may lead to identifying a conservative area in protein family, which is responsible for the protein function. It also creates a possibility of determining a ligand binding site for protein, which is a key issue in drug design. Both experiments were realized as virtual experiments in the GridSpace2 Virtual Laboratory [13] Experiment Workbench [16] and were executed on Zeus cluster provided by PL-Grid.

Tomasz Jadczyk, Maciej Malawski, Marian Bubak, Irena Roterman

InSilicoLab – Managing Complexity of Chemistry Computations

Abstract

InSilicoLab is an application portal designed to support in silico experiments by easily running computational software on Grids. Unlike manual job submission or grid portals, InSilicoLab enables its users to run complex computations without technical knowledge of how to operate the grid resources. Instead, the users may focus only on the information and activities relevant to their research. This paper is a result of a feasibility study of applying the InSilicoLab concept to the domain of computational chemistry. It, therefore, includes a study of chemistry computations and their realisation in InSilicoLab, as well as a description of the generic architecture of the environment.

Joanna Kocot, Tomasz Szepieniec, Daniel Harężlak, Klemens Noga, Mariusz Sterzel

Ab Initio Molecular Dynamics Simulations of Ketocyanine Dyes in Organic Solvents

Abstract

Ab initio Molecular Dynamics simulations of a ketocyanine dye in explicit solvents have been performed on a GP GPU Nvidia accelerators. The effects of single, double or dynamic precision used in calculations has been discussed. Accumulated Molecular Dynamics trajectories have been analyzed with the focus on orbital energies relevant for absorption spectra.

Andrzej Eilmes

Polish Contribution to the Worldwide LHC Computing

Abstract

The computing requirements of LHC experiments, as well as their computing models, are briefly presented. The origin of grid technology and its development in high energy community is outlined, including the Polish participation. The LHC Computing Grid project and its successor, the Worldwide LHC Computing Grid, are presented, including the summary of its successful operations in the first years of LHC data gathering. Against such a background, the creation and operation of the Polish Tier-2 is described, including examples of its use by the LHC experiments.

Artur Binczewski, Michał Bluj, Antoni Cyz, Michał Dwużnik, Maciej Filocha, Łukasz Flis, Ryszard Gokieli, Jarosław Iwaszkiewicz, Marek Kowalski, Patryk Lasoń, Rafał Lichwała, Michał Łopuszyński, Marek Magryś, Piotr Malecki, Norbert Meyer, Krzysztof Nawrocki, Andrzej Olszewski, Andrzej Oziębło, Adam Padée, Henryk Pałka, Marcin Pospieszny, Marcin Radecki, Radosław Rowicki, Dorota Stojda, Marcin Stolarek, Tomasz Szepieniec, Tadeusz Szymocha, Michał Turała, Karol Wawrzyniak, Wojciech Wiślicki, Mariusz Witek, Paweł Wolniewicz

PL-Grid e-Infrastructure for the Cherenkov Telescope Array Observatory

Abstract

We present results of Monte Carlo simulations for the Cherenkov Telescope Array – the next generation observatory of very high energy gamma rays. These calculations are intended to verify design concept for various elements of the array: single telescope design, camera electronics, array configuration, camera and array trigger, etc. The simulations consist of two parts: simulation of the distribution of Cherenkov photons from atmospheric air showers and calculation of response of the telescopes during the detection process. In design study phase of the experiment a large amount of simulations of air showers has been performed with the use of EGEE grid infrastructure, especially centers in Lyon and ACC Cyfronet AGH. Currently we are investigating the details of design of a small telescope – one of several kinds intended for the array.

Anna Barnacka, Leszek Bogacz, Michał Gochna, Mateusz Janiak, Nukri Komin, Giovanni Lamanna, Rafał Moderski, Małgorzata Siudek

Training in the PL-Grid as Key Component to Attract Users to Grid e-Infrastructures

Abstract

This chapter provides an overview of the training facilities in the PL-Grid project. It is well known that procedures necessary to obtain certificates and then install software or get access to the User Interfaces are complicated and due to security reasons cannot be simplified. For these purposes we have crated extended training facilities which allow to get easy access to the Grid. We have installed dedicated infrastructure including the computational resources, certificate authority, user portal and an on-line training system. The efficiency of the training infrastructure, materials and procedures has been proved in the numerous training sessions.

Marcelina Borcz, Krzysztof Benedyczak, Adam Padée, Rafał Kluszczyński, Grzegorz Marczak, Mirosław Zdybek, Maciej Pawlik, Maciej Filocha, Piotr Bała

Dissemination Activities Conducted within the PL-Grid Project as a Successful Means of Promotion of Its Offer and Achievements

Abstract

The dissemination of information as well as promotion of results are the key activities for a scientific project co-funded by European funds. This paper will summarize the dissemination-related activities conducted within the PL-Grid project. It will also point a number of crucial elements of this work – namely, roles and responsibilities of the Project partners, methods of communication, structure of the dissemination efforts and means and measures of their success.

Robert Pająk, Zofia Mosurska

Backmatter