SciLightning: A Cloud Provenance-Based Event Notification for Parallel Workflows

Conducting scientific experiments modeled as workflows is a challenging task due to the complex management of several (often inter-related) computer-based simulations. Many of these scientific workflows are compute intensive and demand High Performance Computing environments to run, such as virtual parallel machines in a cloud computing environment. These workflows commonly present long-term "black-box" executions (

i.e.

several days or weeks), thus making it very difficult for scientists to monitor its execution course. We present a workflow event notification mechanism based on runtime monitoring of provenance data produced by parallel scientific workflow systems in clouds. This mechanism queries provenance data generated at runtime for identifying preconfigured events and notifying scientists using technologies such as Android devices and message services in social networks such as Twitter. The proposed mechanism, named SciLightning, was evaluated by monitoring SciPhy, a large-scale parallel execution of a bioinformatics phylogenetic analysis workflow. SciPhy took six days to complete its execution in Amazon AWS cloud environment using a cloud parallel workflow engine called SciCumulus. The evaluation showed that the proposed approach is effective with respect to monitoring and notifying preconfigured events.