With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific ...applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. In this paper, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis.
•An architecture to collect end-to-end workflow statistics in near real time•The architecture is based on Pegasus WMS and well established tools•A well defined way to deploy the architecture is offered via Docker•Experimental validation done using two real world applications•The experiments where conducted at ExoGENI and Cori supercomputer (NERSC)•Open access data repository containing experimental traces
The future of scientific workflows Deelman, Ewa; Peterka, Tom; Altintas, Ilkay ...
International journal of high performance computing applications/The international journal of high performance computing applications,
01/2018, Volume:
32, Issue:
1
Journal Article
Peer reviewed
Open access
Today’s computational, experimental, and observational sciences rely on computations that involve many related tasks. The success of a scientific mission often hinges on the computer automation of ...these workflows. In April 2015, the US Department of Energy (DOE) invited a diverse group of domain and computer scientists from national laboratories supported by the Office of Science, the National Nuclear Security Administration, from industry, and from academia to review the workflow requirements of DOE’s science and national security missions, to assess the current state of the art in science workflows, to understand the impact of emerging extreme-scale computing systems on those workflows, and to develop requirements for automated workflow management in future and existing environments. This article is a summary of the opinions of over 50 leading researchers attending this workshop. We highlight use cases, computing systems, workflow needs and conclude by summarizing the remaining challenges this community sees that inhibit large-scale scientific workflows from becoming a mainstream tool for extreme-scale science.
The increasing growth of data volume, and the consequent explosion in demand for computational power, are affecting scientific computing, as shown by the rise of extreme data scientific workflows. As ...the need for computing power increases, quantum computing has been proposed as a way to deliver it. It may provide significant theoretical speedups for many scientific applications (i.e., molecular dynamics, quantum chemistry, combinatorial optimization, and machine learning). Therefore integrating quantum computers into the computing continuum constitutes a promising way to speed up scientific computation. However, the scientific computing community still lacks the necessary tools and expertise to fully harness the power of quantum computers in the execution of complex applications such as scientific workflows. In this work, we describe the main characteristics of quantum computing and its main benefits for scientific applications, then we formalize hybrid quantum–classical workflows, explore how to identify quantum components and map them onto resources. We demonstrate concepts on a real use case and define a software architecture for a hybrid workflow management system.
•Formal definition of hybrid quantum–classical systems and workflows;•Finding execution steps for classical workflows on hybrid quantum–classical systems•Study of a hybrid quantum–classical molecular dynamics workflow use case•Research gaps and open challenges for the hybrid quantum–classical workflows.
Workflow applications in science and engineering have steadily increased in variety and scale. Coinciding with this increase has been the relentless effort to improve the performance of these ...applications through exploiting the abundance of resources in hyper-scale clouds and with little attention to resources efficiency. The inefficient use of resources when executing scientific workflows results from both the excessive amount of resources provisioned and the wastage from unused resources among task runs. In this paper, we address the problem of resource-efficient workflow scheduling. To this end, we present the Maximum Effective Reduction (MER) algorithm, a resource efficiency solution that optimizes the resource usage of a workflow schedule generated by any particular scheduling algorithm. MER trades the minimal makespan increase for the maximal resource usage reduction by consolidating tasks with the exploitation of resource inefficiency in the original workflow schedule. The main novelty of MER lies in its identification of “near-optimal” trade-off point between makespan increase and resource usage reduction. Finding such a point is of great practical importance and can lead to: (1) improvements in resource utilization, (2) reductions in resource provisioning, and (3) savings in energy consumption. Another significant contribution of this work is MER’s broad applicability. In essence, MER can be applied to any environments that deal with the execution of (scientific) workflows of many precedence-constrained tasks although MER best suits for the IaaS cloud model. Based on results obtained from our extensive simulations using scientific workflow traces, we demonstrate MER is capable of reducing the amount of actual resources used by 54% with an average makespan increase of less than 10%. The efficacy of MER is further verified by results (from a comprehensive set of experiments with varying makespan delay limits) that show the resource usage reduction, makespan increase and the trade-off between them for various workflow applications.
Cloud computing, a distributed computing paradigm, enables delivery of IT resources over the Internet and follows the pay-as-you-go billing model. Workflow scheduling is one of the most challenging ...problems in cloud computing. Although, workflow scheduling on distributed systems like grids and clusters have been extensively studied, however, these solutions are not viable for a cloud environment. It is because, a cloud environment differs from other distributed environment in two major ways: on-demand resource provisioning and pay-as-you-go pricing model. Thus, to achieve the true benefits of workflow orchestration onto cloud resources novel approaches that can capitalize the advantages and address the challenges specific to a cloud environment needs to be developed. This work proposes a dynamic cost-effective deadline-constrained heuristic algorithm for scheduling a scientific workflow in a public cloud. The proposed technique aims to exploit the advantages offered by cloud computing while taking into account the virtual machine (VM) performance variability and instance acquisition delay to identify a just-in-time schedule of a deadline constrained scientific workflow at lesser costs. Performance evaluation on some well-known scientific workflows exhibit that the proposed algorithm delivers better performance in comparison to the current state-of-the-art heuristics.
The paper presents a platform for distributed computing, developed using the latest software technologies and computing paradigms to enable big data mining. The platform, called ClowdFlows, is ...implemented as a cloud-based web application with a graphical user interface which supports the construction and execution of data mining workflows, including web services used as workflow components. As a web application, the ClowdFlows platform poses no software requirements and can be used from any modern browser, including mobile devices. The constructed workflows can be declared either as private or public, which enables sharing the developed solutions, data and results on the web and in scientific publications. The server-side software of ClowdFlows can be multiplied and distributed to any number of computing nodes. From a developer’s perspective the platform is easy to extend and supports distributed development with packages. The paper focuses on big data processing in the batch and real-time processing mode. Big data analytics is provided through several algorithms, including novel ensemble techniques, implemented using the map-reduce paradigm and a special stream mining module for continuous parallel workflow execution. The batch mode and real-time processing mode are demonstrated with practical use cases. Performance analysis shows the benefit of using all available data for learning in distributed mode compared to using only subsets of data in non-distributed mode. The ability of ClowdFlows to handle big data sets and its nearly perfect linear speedup is demonstrated.
•We present a cloud based platform for big data and stream processing with workflows.•The ClowdFlows platform enables processing of multiple concurrent data streams.•Several machine learning algorithms were implemented in the map-reduce paradigm.•Using all data in distributed mode is better than using a subset in non-distributed.•The ClowdFlows platform handles big data sets with nearly perfect linear speedup.
In the recent years, the use of workflows has significantly expanded from its original domain of business processes towards new areas. The increasing demand for individual and more flexible workflows ...asks for new methods that support domain experts to create, monitor, and adapt workflows. The emergent field of process-oriented case-based reasoning addresses this problem by proposing methods for reasoning with workflows based on experience. New workflows can be constructed by reuse of already available similar workflows from a repository. Hence, methods for the similarity assessment of workflows and for the efficient retrieval of similar workflows from a repository are of core importance. To this end, we describe a new generic model for representing workflows as semantically labeled graphs, together with a related model for knowledge intensive similarity measures. Further, new algorithms for workflow similarity computation, based on A⁎ search are described. A new retrieval algorithm is introduced that goes beyond traditional sequential retrieval for graphs, interweaving similarity computation with case selection. We describe the application of this model and several experimental evaluations of the algorithms in the domain of scientific workflows and in the domain of business workflows, thereby showing its broad applicability.
The elasticity of Cloud infrastructures makes them a suitable platform for execution of deadline-constrained workflow applications, because resources available to the application can be dynamically ...increased to enable application speedup. Existing research in execution of scientific workflows in Clouds either try to minimize the workflow execution time ignoring deadlines and budgets or focus on the minimization of cost while trying to meet the application deadline. However, they implement limited contingency strategies to correct delays caused by underestimation of tasks execution time or fluctuations in the delivered performance of leased public Cloud resources. To mitigate effects of performance variation of resources on soft deadlines of workflow applications, we propose an algorithm that uses idle time of provisioned resources and budget surplus to replicate tasks. Simulation experiments with four well-known scientific workflows show that the proposed algorithm increases the likelihood of deadlines being met and reduces the total execution time of applications as the budget available for replication increases.
In-situ workflows are a particular class of scientific workflows where different components (such as simulation, visualization, machine learning, and data analysis) run concurrently. In an in-situ ...workflow, data continuously flows between components during the whole workflow lifetime in a pipeline fashion. The overall throughput of an in-situ workflow is limited by the slowest moving part of the workflow: either a time-consuming computation component or an expensive data transfer operation between different components. Given geographically distributed heterogeneous computing environments and an in-situ workflow comprising various component applications, it is challenging to produce optimized deployment plans to decide where to launch each component application and how much computing resources to allocate for each component. In this work, we first formulate and define the in-situ workflow planning problem and then discuss the uniqueness of the problem. We design a heuristic-based solver based on our new “Scheduled-Neighbors-Lookup” (SNL) algorithm, which produces effective deployment plans with much less time, compared with the mathematical-optimization-based solver using CPLEX. Our experiments with both synthetic and real-world workflows show that the SNL algorithm can find optimized solutions whose qualities are comparable to that of the CPLEX optimization method with significantly reduced problem-solving time (e.g., more than 20,000 times faster on average for in-situ workflows with 14 components). Compared with existing methods such as HEFT, R-Storm, and T3 schedulers, the SNL algorithm can generate resource plans with higher throughput, with increased time and resource efficiency.
•Problem formulation of resource planning for complex in-situ workflows.•Consideration of geo-distributed heterogeneous computing environments.•A fast and effective heuristic-based solver based on the pipelined execution pattern.•Experiments with performance analyses for both synthetic and real-world workloads.
PDF
