The Production and Distributed Analysis system (PanDA), used for workload management in the ATLAS Experiment at the LHC for over a decade, has in recent years expanded its reach to diverse new ...resource types such as HPCs, and innovative new workflows such as the Event Service. PanDA meets the heterogeneous resources it harvests in the PanDA Pilot, which has embarked on a next-generation reengineering to efficiently integrate and exploit the new platforms and workflows. The new modular architecture is the product of a year of design and prototyping in conjunction with the design of a completely new component, Harvester, that will mediate a richer flow of control and information between Pilot and PanDA. Harvester will enable more intelligent and dynamic matching between processing tasks and resources, with an initial focus on HPCs, simplifying the operator and user view of a PanDA site but internally leveraging deep information gathering on the resource to accrue detailed knowledge of a site's capabilities and dynamic state to inform the matchmaking. This paper will give an overview of the new Pilot architecture, how it will be used in and beyond ATLAS, its relation to Harvester, and the work ahead.
The LHC experiments are preparing for the precision measurements and further discoveries that will be made possible by higher LHC energies from April 2015 (LHC Run2). The need for simulation, data ...processing and analysis would overwhelm the expected capacity of grid infrastructure computing facilities deployed by the Worldwide LHC Computing Grid (WLCG). To meet this challenge the integration of the opportunistic resources into LHC computing model is highly important. The Tier-1 facility at Kurchatov Institute (NRC-KI) in Moscow is a part of WLCG and it will process, simulate and store up to 10% of total data obtained from ALICE, ATLAS and LHCb experiments. In addition Kurchatov Institute has supercomputers with peak performance 0.12 PFLOPS. The delegation of even a fraction of supercomputing resources to the LHC Computing will notably increase total capacity. In 2014 the development a portal combining a Tier-1 and a supercomputer in Kurchatov Institute was started to provide common interfaces and storage. The portal will be used not only for HENP experiments, but also by other data- and compute-intensive sciences like biology with genome sequencing analysis; astrophysics with cosmic rays analysis, antimatter and dark matter search, etc.
Creation of global e-Infrastructure involves an integration of isolated local resources into common heterogeneous computing environment. In 2014 a pioneering work to develop a large scale data- and ...task- management system for federated heterogeneous resources has been started at the National Research Centre “Kurchatov Institute” (NRC KI, Moscow). As a part of this work, we have designed, developed and deployed a portal to submit payloads to heterogeneous computing infrastructure. It combines Tier-1, Cloud-infrastructure, and a supercomputer at the Kurchatov institute. This portal is aimed to provide a common interface to submit tasks to Grid sites, commercial and academic clouds and supercomputers. Integration of Tier-1 and the supercomputer has allowed to notably increase total CPU capacity available for Large Hadron Collider (LHC) experiments. The portal can be used not only for High Energy Physics (HEP) applications, but also for other compute-intensive sciences such as bioinformatics with genome and sequence analysis; astrophysics with cosmic rays analysis, antimatter and dark matter search, etc.
This article describes developed portal as a top layer for computing facilities infrastructure for High Energy Physics and other compute-intensive science applications. The article presents the results of using PanDA at NRC KI supercomputer/Cloud as underlying technology for the portal.
PDF
