The AMGA Metadata Service Koblitz, B.; Santos, N.; Pose, V.
Journal of grid computing,
03/2008, Volume:
6, Issue:
1
Journal Article
We present the AMGA metadata catalogue, which was developed as part of the EGEE (enabling Grids for EsciencE) project’s gLite Grid middleware. AMGA provides access to meta data for files stored on ...the Grid, as well as a simplified general access to relational data stored in database systems. Design and implementation of AMGA was done in close collaboration with the very diverse EGEE user community to make sure all functionality, performance and security requirements were met. In particular, AMGA targets the needs of the high energy physics community to rapidly access very large amounts of metadata, as well as the needs for security of the biomedical community. AMGA therefore tightly integrates fine grained access control making use of a virtual organisation management system. In addition, it offers advanced federation and features to increase dependability, performance and data security.
The ATLAS detector at CERN's Large Hadron Collider presents data handling requirements on an unprecedented scale. From 2008 on the ATLAS distributed data management system, Don Quijote2 (DQ2), must ...manage tens of petabytes of experiment data per year, distributed globally via the LCG, OSG and NDGF computing grids, now commonly known as the WLCG. Since its inception in 2005 DQ2 has continuously managed all experiment data for the ATLAS collaboration, which now comprises over 3000 scientists participating from more than 150 universities and laboratories in 34 countries. Fulfilling its primary requirement of providing a highly distributed, fault-tolerant and scalable architecture DQ2 was successfully upgraded from managing data on a terabyte-scale to managing data on a petabyte-scale. We present improvements and enhancements to DQ2 based on the increasing demands for ATLAS data management. We describe performance issues, architectural changes and implementation decisions, the current state of deployment in test and production as well as anticipated future improvements. Test results presented here show that DQ2 is capable of handling data up to and beyond the requirements of full-scale data-taking.
We explore the feasibility of implementing a reliable, high performance, distributed storage system on a commodity computing cluster. Files are distributed across storage nodes using erasure coding ...with small low-density parity-check (LDPC) codes, which provide high-reliability with small storage and performance overhead. We present performance measurements done on a prototype system comprising 50 nodes, which are self organised using a peer-to-peer overlay.
Metadata services on the Grid Santos, Nuno; Koblitz, Birger
Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment,
04/2006, Volume:
559, Issue:
1
Journal Article
Peer reviewed
Open access
We present an interface for metadata access on the Grid, designed to support flexible schema management, efficient retrieval of large result sets and to allow a broad range of implementations. We ...also describe an implementation of this interface, which supports a wide range of storage back-ends and two access protocols: SOAP and a TCP-streaming-based protocol. This interface and implementation have been selected as the official metadata components of the gLite-EGEE middleware. Finally, we present the results of extensive performance studies, where the two front-ends are compared to evaluate the cost of using SOAP as metadata access protocol.
Performance comparison of the LCG2 and gLite file catalogues Munro, C.; Koblitz, B.
Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment,
04/2006, Volume:
559, Issue:
1
Journal Article
Peer reviewed
When the Large Hadron Collider (LHC) begins operation at CERN in 2007 it will produce data in volumes never before seen. The LHC Computing Grid will provide the middleware necessary to distribute and ...analyse petabytes of this data. One of the critical factors in the smooth running of this system is the performance of the file catalogues which translate logical filenames into physical locations. We present a detailed study comparing the LCG project's catalogue with the gLite FiReMan catalogue developed in the EGEE project, covering performance and respective merits and shortcomings.
When the Large Hadron Collider (LHC) begins operation at CERN in 2007 it will produce data in volumes never before seen. Physicists around the world will manage, distribute and analyse petabytes of ...this data using the middleware provided by the LHC Computing Grid. One of the critical factors in the smooth running of this system is the performance of the file catalogues which allow users to access their files with a logical filename without knowing their physical location. This paper presents a detailed study comparing the performance and respective merits and shortcomings of two of the main catalogues: the LCG File Catalogue and the gLite FiReMan catalogue
The medical community is producing and manipulating a tremendous volume of digital data for which computerized archiving, processing and analysis is needed. Grid infrastructures are promising for ...dealing with challenges arising in computerized medicine but the manipulation of medical data on such infrastructures faces both the problem of interconnecting medical information systems to Grid middlewares and of preserving patients’ privacy in a wide and distributed multi-user system. These constraints are often limiting the use of Grids for manipulating sensitive medical data. This paper describes our design of a medical data management system taking advantage of the advanced gLite data management services, developed in the context of the EGEE project, to fulfill the stringent needs of the medical community. It ensures medical data protection through strict data access control, anonymization and encryption. The multi-level access control provides the flexibility needed for implementing complex medical use-cases. Data anonymization prevents the exposure of most sensitive data to unauthorized users, and data encryption guarantees data protection even when it is stored at remote sites. Moreover, the developed prototype provides a Grid storage resource manager (SRM) interface to standard medical DICOM servers thereby enabling transparent access to medical data without interfering with medical practice.
Migration of ATLAS PanDA to CERN Stewart, Graeme Andrew; Klimentov, Alexei; Koblitz, Birger ...
Journal of physics. Conference series,
04/2010, Volume:
219, Issue:
6
Journal Article
Peer reviewed
Open access
The ATLAS Production and Distributed Analysis System (PanDA) is a key component of the ATLAS distributed computing infrastructure. All ATLAS production jobs, and a substantial amount of user and ...group analysis jobs, pass through the PanDA system, which manages their execution on the grid. PanDA also plays a key role in production task definition and the data set replication request system. PanDA has recently been migrated from Brookhaven National Laboratory (BNL) to the European Organization for Nuclear Research (CERN), a process we describe here. We discuss how the new infrastructure for PanDA, which relies heavily on services provided by CERN IT, was introduced in order to make the service as reliable as possible and to allow it to be scaled to ATLAS's increasing need for distributed computing. The migration involved changing the backend database for PanDA from MySQL to Oracle, which impacted upon the database schemas. The process by which the client code was optimised for the new database backend is discussed. We describe the procedure by which the new database infrastructure was tested and commissioned for production use. Operations during the migration had to be planned carefully to minimise disruption to ongoing ATLAS offline computing. All parts of the migration were fully tested before commissioning the new infrastructure and the gradual migration of computing resources to the new system allowed any problems of scaling to be addressed.
PDF
