E-resources
-
Papadimitriou, George; Wang, Cong; Vahi, Karan; da Silva, Rafael Ferreira; Mandal, Anirban; Liu, Zhengchun; Mayani, Rajiv; Rynge, Mats; Kiran, Mariam; Lynch, Vickie E.; Kettimuthu, Rajkumar; Deelman, Ewa; Vetter, Jeffrey S.; Foster, Ian
Future generation computer systems, April 2021, 2021-04-00, 2021-04-01, Volume: 117, Issue: CJournal Article
With the increased prevalence of employing workflows for scientific computing and a push towards exascale computing, it has become paramount that we are able to analyze characteristics of scientific applications to better understand their impact on the underlying infrastructure and vice-versa. Such analysis can help drive the design, development, and optimization of these next generation systems and solutions. In this paper, we present the architecture, integrated with existing well-established and newly developed tools, to collect online performance statistics of workflow executions from various, heterogeneous sources and publish them in a distributed database (Elasticsearch). Using this architecture, we are able to correlate online workflow performance data, with data from the underlying infrastructure, and present them in a useful and intuitive way via an online dashboard. We have validated our approach by executing two classes of real-world workflows, both under normal and anomalous conditions. The first is an I/O-intensive genome analysis workflow; the second, a CPU- and memory-intensive material science workflow. Based on the data collected in Elasticsearch, we are able to demonstrate that we can correctly identify anomalies that we injected. The resulting end-to-end data collection of workflow performance data is an important resource of training data for automated machine learning analysis. •An architecture to collect end-to-end workflow statistics in near real time•The architecture is based on Pegasus WMS and well established tools•A well defined way to deploy the architecture is offered via Docker•Experimental validation done using two real world applications•The experiments where conducted at ExoGENI and Cori supercomputer (NERSC)•Open access data repository containing experimental traces
![loading ... loading ...](themes/default/img/ajax-loading.gif)
Shelf entry
Permalink
- URL:
Impact factor
Access to the JCR database is permitted only to users from Slovenia. Your current IP address is not on the list of IP addresses with access permission, and authentication with the relevant AAI accout is required.
Year | Impact factor | Edition | Category | Classification | ||||
---|---|---|---|---|---|---|---|---|
JCR | SNIP | JCR | SNIP | JCR | SNIP | JCR | SNIP |
Select the library membership card:
If the library membership card is not in the list,
add a new one.
DRS, in which the journal is indexed
Database name | Field | Year |
---|
Links to authors' personal bibliographies | Links to information on researchers in the SICRIS system |
---|
Source: Personal bibliographies
and: SICRIS
The material is available in full text. If you wish to order the material anyway, click the Continue button.