The ATLAS Nightly Build System is a major component in the ATLAS collaborative software organization, validation, and code approval scheme. For over 10 years of development it has evolved into a ...factory for automatic release production and grid distribution. The 50 multi-platform branches of ATLAS releases provide vast opportunities for testing new packages, verification of patches to existing software, and migration to new platforms and compilers for ATLAS code that currently contains 2200 packages with 4 million C++ and 1.4 million python scripting lines written by about 1000 developers. Recent development was focused on the integration of ATLAS Nightly Build and Installation systems. The nightly releases are distributed and validated and some are transformed into stable releases used for data processing worldwide. The ATLAS Nightly System is managed by the NICOS control tool on a computing farm with 50 powerful multiprocessor nodes. NICOS provides the fully automated framework for the release builds, testing, and creation of distribution kits. The ATN testing framework of the Nightly System runs unit and integration tests in parallel suites, fully utilizing the resources of multi-core machines, and provides the first results even before compilations complete. The NICOS error detection system is based on several techniques and classifies the compilation and test errors according to their severity. It is periodically tuned to place greater emphasis on certain software defects by highlighting the problems on NICOS web pages and sending automatic e-mail notifications to responsible developers. These and other recent developments will be presented and future plans will be described.
The offline software of the ATLAS experiment at the Large Hadron Collider (LHC) serves as the platform for detector data reconstruction, simulation and analysis. It is also used in the detector's ...trigger system to select LHC collision events during data taking. The ATLAS offline software consists of several million lines of C++ and Python code organized in a modular design of more than 2000 specialized packages. Because of different workflows, many stable numbered releases are in parallel production use. To accommodate specific workflow requests, software patches with modified libraries are distributed on top of existing software releases on a daily basis. The different ATLAS software applications also require a flexible build system that strongly supports unit and integration tests. Within the last year this build system was migrated to CMake. A CMake configuration has been developed that allows one to easily set up and build the above mentioned software packages. This also makes it possible to develop and test new and modified packages on top of existing releases. The system also allows one to detect and execute partial rebuilds of the release based on single package changes. The build system makes use of CPack for building RPM packages out of the software releases, and CTest for running unit and integration tests. We report on the migration and integration of the ATLAS software to CMake and show working examples of this large scale project in production.
The ATLAS software infrastructure facilitates efforts of more than 1000 developers working on the code base of 2200 packages with 4 million lines of C++ and 1.4 million lines of python code. The ...ATLAS offline code management system is the powerful, flexible framework for processing new package versions requests, probing code changes in the Nightly Build System, migration to new platforms and compilers, deployment of production releases for worldwide access and supporting physicists with tools and interfaces for efficient software use. It maintains multi-stream, parallel development environment with about 70 multi-platform branches of nightly releases and provides vast opportunities for testing new packages, for verifying patches to existing software and for migrating to new platforms and compilers. The system evolution is currently aimed on the adoption of modern continuous integration (CI) practices focused on building nightly releases early and often, with rigorous unit and integration testing. This paper describes the CI incorporation program for the ATLAS software infrastructure. It brings modern open source tools such as Jenkins and GitLab into the ATLAS Nightly System, rationalizes hardware resource allocation and administrative operations, provides improved feedback and means to fix broken builds promptly for developers. Once adopted, ATLAS CI practices will improve and accelerate innovation cycles and result in increased confidence in new software deployments. The paper reports the status of Jenkins integration with the ATLAS Nightly System as well as short and long term plans for the incorporation of CI practices.
In the last year ATLAS has radically updated its software development infrastructure hugely reducing the complexity of building releases and greatly improving build speed, flexibility and code ...testing. The first step in this transition was the adoption of CMake as the software build system over the older CMT. This required the development of an automated translation from the old system to the new, followed by extensive testing and improvements. This resulted in a far more standard build process that was married to the method of building ATLAS software as a series of 12 separate projects from Subversion. We then proceeded with a migration of the code base from Subversion to Git. As the Subversion repository had been structured to manage each package more or less independently there was no simple mapping that could be used to manage the migration into Git. Instead a specialist set of scripts that captured the software changes across official software releases was developed. With some clean up of the repository and the policy of only migrating packages in production releases, we managed to reduce the repository size from 62 GiB to 220 MiB. After moving to Git we took the opportunity to introduce continuous integration so that now each code change from developers is built and tested before being approved. With both CMake and Git in place we also dramatically simplified the build management of ATLAS software. Many heavyweight homegrown tools were dropped and the build procedure was reduced to a single bootstrap of some external packages, followed by a full build of the rest of the stack. This has reduced the time for a build by a factor of 2. It is now easy to build ATLAS software, freeing developers to test compile intrusive changes or new platform ports with ease. We have also developed a system to build lightweight ATLAS releases, for simulation, analysis or physics derivations which can be built from the same branch.
ATLAS Nightly Build System Upgrade Dimitrov, G; Obreshkov, E; Simmons, B ...
Journal of physics. Conference series,
01/2014, Letnik:
513, Številka:
5
Journal Article
Recenzirano
Odprti dostop
The ATLAS Nightly Build System is a facility for automatic production of software releases. Being the major component of ATLAS software infrastructure, it supports more than 50 multi-platform ...branches of nightly releases and provides ample opportunities for testing new packages, for verifying patches to existing software, and for migrating to new platforms and compilers. The Nightly System testing framework runs several hundred integration tests of different granularity and purpose. The nightly releases are distributed and validated, and some are transformed into stable releases used for data processing worldwide. The first LHC long shutdown (2013-2015) activities will elicit increased load on the Nightly System as additional releases and builds are needed to exploit new programming techniques, languages, and profiling tools. This paper describes the plan of the ATLAS Nightly Build System Long Shutdown upgrade. It brings modern database and web technologies into the Nightly System, improves monitoring of nightly build results, and provides new tools for offline release shifters. We will also outline our long-term plans for distributed nightly releases builds and testing.
The ATLAS collaboration is managing one of the largest collections of software among the High Energy Physics experiments. Traditionally, this software has been distributed via rpm or pacman packages, ...and has been installed in every site and user's machine, using more space than needed since the releases share common files but are installed in their own trees. As soon as the software has grown in size and number of releases this approach showed its limits, in terms of manageability, used disk space and performance. The adopted solution is based on the CernVM File System, a fuse-based HTTP, read-only filesystem which guarantees file de-duplication, on-demand file transfer with caching, scalability and performance. Here we describe the ATLAS experience in setting up the CVMFS facility and putting it into production, for different type of use-cases, ranging from single users’ machines up to large data centers, for both software and conditions data. The performance of CernVM-FS, both with software and condition data access, will be shown, comparing with other filesystems currently in use by the collaboration.
The automated multi-platform software nightly build system is a major component in the ATLAS collaborative software organization, validation and code approval schemes. Code developers from ATLAS ...participating Institutes spread all around the world use about 30 branches of nightly releases for testing new packages, verification of patches to existing software, and migration to new platforms and compilers. The nightly releases lead up to, and are the basis of, stable software releases used for data processing worldwide. The ATLAS nightly builds are managed by the fully automated NICOS framework on the computing farm with 44 powerful multiprocessor nodes. The ATN test tool is embedded within the nightly system and provides results shortly after full compilations complete. Other test frameworks are synchronized with NICOS jobs and run larger scale validation jobs using the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback, e-mail notifications about nightly releases problems are automatically distributed to the developers responsible.
We update our CHEP06 2 presentation on the ATLAS experiment software infrastructure used to build, validate, distribute, and document the ATLAS offline software. The ATLAS collaboration's ...computational resources and software developers are distributed around the globe in about 35 counties. The ATLAS offline code base is currently over 7 million source lines of code in 10,000+ C++ classes organized into about 2,000 packages. More than 400 developers contribute code each month. Since our last report, we have developed a powerful, flexible system to request code versions to be included in software builds, made changes to our software building tools, increased the number of daily builds used to validate significant code changes, improved the tools for distributing the code to our computational sites around the world, and made many advancements in the tools to document the code.
Linear colliders offer a unique opportunity to study γγ and γe interactions. Using the laser backscattering method one can obtain γγ, γe colliding beams with an energy and luminosity comparable to ...that in e
+e
− collisions. This work is part of the Conceptual Design of TESLA/SBLC linear colliders considering a second interaction region for γγ and γe collisions. We consider here possible physics in high-energy γγ, γe collisions, e→γ conversion, requirements to lasers, collision schemes, attainable luminosities, backgrounds, possible lasers, optics at the interaction region and other associated problems.
PDF
