DIRAC (Distributed Infrastructure with Remote Agent Control) is a general framework for the management of tasks over distributed heterogeneous computing environments. It has been originally developed ...to support the production activities of the LHCb (Large Hadron Collider Beauty) experiment and today is extensively used by several particle physics and biology communities. Current (Fermi Large Area Telescope – LAT) and planned (Cherenkov Telescope Array – CTA) new generation astrophysical/cosmological experiments, with very large processing and storage needs, are currently investigating the usability of DIRAC in this context. Each of these use cases has some peculiarities: Fermi-LAT will interface DIRAC to its own workflow system to allow the access to the grid resources, while CTA is using DIRAC as workflow management system for Monte Carlo production and analysis on the grid. We describe the prototype effort that we lead toward deploying a DIRAC solution for some aspects of Fermi-LAT and CTA needs.
DIRAC, the LHCb community Grid solution, was considerably reengineered in order to meet all the requirements for processing the data coming from the LHCb experiment. It is covering all the tasks ...starting with raw data transportation from the experiment area to the grid storage, data processing up to the final user analysis. The reengineered DIRAC3 version of the system includes a fully grid security compliant framework for building service oriented distributed systems; complete Pilot Job framework for creating efficient workload management systems; several subsystems to manage high level operations like data production and distribution management. The user interfaces of the DIRAC3 system providing rich command line and scripting tools are complemented by a full-featured Web portal providing users with a secure access to all the details of the system status and ongoing activities. We will present an overview of the DIRAC3 architecture, new innovative features and the achieved performance. Extending DIRAC3 to manage computing resources beyond the WLCG grid will be discussed. Experience with using DIRAC3 by other user communities than LHCb and in other application domains than High Energy Physics will be shown to demonstrate the general-purpose nature of the system.
The Cherenkov Telescope Array (CTA) - an array of many tens of Imaging Atmospheric Cherenkov Telescopes deployed on an unprecedented scale - is the next generation instrument in the field of very ...high energy gamma-ray astronomy. CTA will operate as an open observatory providing data products to the scientific community. An average data stream of about 10 GB s for about 1000 hours of observation per year, thus producing several PB year, is expected. Large CPU time is required for data-processing as well for massive Monte Carlo simulations needed for detector calibration purposes. The current CTA computing model is based on a distributed infrastructure for the archive and the data off-line processing. In order to manage the off-line data-processing in a distributed environment, CTA has evaluated the DIRAC (Distributed Infrastructure with Remote Agent Control) system, which is a general framework for the management of tasks over distributed heterogeneous computing environments. In particular, a production system prototype has been developed, based on the two main DIRAC components, i.e. the Workload Management and Data Management Systems. After three years of successful exploitation of this prototype, for simulations and analysis, we proved that DIRAC provides suitable functionalities needed for the CTA data processing. Based on these results, the CTA development plan aims to achieve an operational production system, based on the DIRAC Workload Management System, to be ready for the start of CTA operation phase in 2017-2018. One more important challenge consists of the development of a fully automatized execution of the CTA workflows. For this purpose, we have identified a third DIRAC component, the so-called Transformation System, which offers very interesting functionalities to achieve this automatisation. The Transformation System is a 'data-driven' system, allowing to automatically trigger data-processing and data management operations according to pre-defined scenarios. In this paper, we present a brief summary of the DIRAC evaluation done so far, as well as the future developments planned for the CTA production system. In particular, we will focus on the developments of CTA automatic workflows, based on the Transformation System. As a result, we also propose some design optimizations of the Transformation System, in order to fully support the most complex workflows, envisaged in the CTA processing.
DIRAC RESTful API Ramo, A Casajus; Diaz, R Graciani; Tsaregorodtsev, A
Journal of physics. Conference series,
01/2012, Letnik:
396, Številka:
5
Journal Article
Recenzirano
Odprti dostop
The DIRAC framework for distributed computing has been designed as a flexible and modular solution that can be adapted to the requirements of any community. Users interact with DIRAC via command ...line, using the web portal or accessing resources via the DIRAC python API. The current DIRAC API requires users to use a python version valid for DIRAC. Some communities have developed their own software solutions for handling their specific workload, and would like to use DIRAC as their back-end to access distributed computing resources easily. Many of these solutions are not coded in python or depend on a specific python version. To solve this gap DIRAC provides a new language agnostic API that any software solution can use. This new API has been designed following the RESTful principles. Any language with libraries to issue standard HTTP queries may use it. GSI proxies can still be used to authenticate against the API services. However GSI proxies are not a widely adopted standard. The new DIRAC API also allows clients to use OAuth for delegating the user credentials to a third party solution. These delegated credentials allow the third party software to query to DIRAC on behalf of the users. This new API will further expand the possibilities communities have to integrate DIRAC into their distributed computing models.
Executor Framework for DIRAC Ramo, A Casajus; Diaz, R Graciani
Journal of physics. Conference series,
01/2012, Letnik:
396, Številka:
5
Journal Article
Recenzirano
Odprti dostop
DIRAC framework for distributed computing has been designed as a group of collaborating components, agents and servers, with persistent database back-end. Components communicate with each other using ...DISET, an in-house protocol that provides Remote Procedure Call (RPC) and file transfer capabilities. This approach has provided DIRAC with a modular and stable design by enforcing stable interfaces across releases. But it made complicated to scale further with commodity hardware. To further scale DIRAC, components needed to send more queries between them. Using RPC to do so requires a lot of processing power just to handle the secure handshake required to establish the connection. DISET now provides a way to keep stable connections and send and receive queries between components. Only one handshake is required to send and receive any number of queries. Using this new communication mechanism DIRAC now provides a new type of component called Executor. Executors process any task (such as resolving the input data of a job) sent to them by a task dispatcher. This task dispatcher takes care of persisting the state of the tasks to the storage backend and distributing them among all the Executors based on the requirements of each task. In case of a high load, several Executors can be started to process the extra load and stop them once the tasks have been processed. This new approach of handling tasks in DIRAC makes Executors easy to replace and replicate, thus enabling DIRAC to further scale beyond the current approach based on polling agents.
The Cherenkov Telescope Array (CTA) - an array of several tens of Cherenkov telescopes - is the next generation of ground-based instrument in the field of very high energy gamma-ray astronomy. The ...CTA observatory is expected to produce a main data stream for permanent storage of the order of 1-to-5 GB/s for about 1000 hours of observation per year, thus producing a total data volume of the order of several PB per year. The CPU time needed to calibrate and process one hour of data taking will be of the order of some thousands CPU hours with current technology. The high data rate of CTA, together with the large computing power requirements for Monte Carlo (MC) simulations, need dedicated computing resources. Massive MC simulations are needed to study the physics of cosmic-ray atmospheric showers as well as telescope response and performance for different detectors and layout configurations. Given these large storage and computing requirements, the Grid approach is well suited, and a vast number of MC simulations are already running on the European Grid Infrastructure (EGI). In order to optimize resource usage and to handle all production and future analysis activities in a coherent way, a high-level framework with advanced functionalities is desirable. For this purpose we have preliminarly evaluated the DIRAC framework for distributed computing and tested it for the CTA workload and data management systems. In this paper we present a possible implementation of a Distributed Computing Infrastructure (DCI) Computing Model for CTA as well as the benchmark test results of DIRAC.
The gUSE (Grid User Support Environment) framework allows to create, store and distribute application workflows. This workflow architecture includes a wide variety of payload execution operations, ...such as loops, conditional execution of jobs and combination of output. These complex multi-job workflows can easily be created and modified by application developers through the WS-PGRADE portal. The portal also allows end users to download and use existing workflows, as well as executing them. The DIRAC framework for distributed computing, a complete Grid solution for a community of users needing access to distributed computing resources, has been integrated into the gUSE/WS-PGRADE system. This integration allows the execution of gUSE workflows in a distributed computing environment, thus greatly expanding the capability of the portal to several Grids and Cloud Computing facilities. The main features and possibilities of the gUSE/WS-PGRADE-DIRAC system, as well as the benefits for users, will be outlined and discussed.
DIRAC: a community grid solution Tsaregorodtsev, A; Bargiotti, M; Brook, N ...
Journal of physics. Conference series,
07/2008, Letnik:
119, Številka:
6
Journal Article
Recenzirano
Odprti dostop
The DIRAC system was developed in order to provide a complete solution for using the distributed computing resources of the LHCb experiment at CERN for data production and analysis. It allows a ...concurrent use of over 10K CPUs and 10M file replicas distributed over many tens of sites. The sites can be part of a Computing Grid such as WLCG or standalone computing clusters all integrated in a single management structure. DIRAC is a generic system with the LHCb specific functionality incorporated through a number of plug-in modules. It can be easily adapted to the needs of other communities. Special attention is paid to the resilience of the DIRAC components to allow an efficient use of non-reliable resources. The DIRAC production management components provide a framework for building highly automated data production systems including data distribution and data driven workload scheduling. In this paper we give an overview of the DIRAC system architecture and design choices. We show how different components are put together to compose an integrated data processing system including all the aspects of the LHCb experiment - from the MC production and raw data reconstruction to the final user analysis.
The LHCb computing data challenge DC06 Nandakumar, R; Jimenez, S G; Adinolfi, M ...
Journal of physics. Conference series,
07/2008, Letnik:
119, Številka:
7
Journal Article
Recenzirano
Odprti dostop
The worldwide computing grid is essential to the LHC experiments in analysing the data collected by the detectors. Within LHCb, the computing model aims to simulate data at Tier-2 grid sites as well ...as non-grid resources. The reconstruction, stripping and analysis of the produced LHCb data will pimarily place at the Tier-1 centres. The computing data challenge DC06 started in May 2006 with the primary aims being to exercise the LHCb computing mod and to produce events which will be used for analyses in the forthcoming LHCb physics book. This paper gives an overview of the LHCb computing model and addresses the challenges and experiences during DC06. The management of the production of Monte Carlo data on the LCG was done using the DIRAC worklad management system which in turn uses the WLCG infrastructure and middleware. We shall report on the amount of data simulated during DC06, including the performance of the sites used. The paper will also summarise the experience gained during DC06, in particular he distribution of data to the Ter-1 sits and the access to this data.
Abstract
Aria is a plant hosting a
$${350}\,\hbox {m}$$
350
m
cryogenic isotopic distillation column, the tallest ever built, which is being installed in a mine shaft at Carbosulcis S.p.A., ...Nuraxi-Figus (SU), Italy. Aria is one of the pillars of the argon dark-matter search experimental program, lead by the Global Argon Dark Matter Collaboration. It was designed to reduce the isotopic abundance of
$${^{39}\hbox {Ar}}$$
39
Ar
in argon extracted from underground sources, called Underground Argon (UAr), which is used for dark-matter searches. Indeed,
$${^{39}\hbox {Ar}}$$
39
Ar
is a
$$\beta $$
β
-emitter of cosmogenic origin, whose activity poses background and pile-up concerns in the detectors. In this paper, we discuss the requirements, design, construction, tests, and projected performance of the plant for the isotopic cryogenic distillation of argon. We also present the successful results of the isotopic cryogenic distillation of nitrogen with a prototype plant.
PDF
