LHCb has been using the CERN/IT developed Conditions Database library COOL for several years, during LHC Run 1 and Run 2. With the opportunity window of the second long shutdown of LHC, in ...preparation for Run 3 and the upgraded LHCb detector, we decided to investigate alternatives to COOL as Conditions Database backend. In particular, given our conditions and detector description data model, we investigated the possibility of reusing the internal Git repository database as a conditions storage, and we adopted it since 2017 data taking. The adoption of Git gave us improved performance, smaller storage size and simplified maintenance and deployment.
In this paper we describe the implementation of our Git Conditions Database and the way it simplified our detector description and conditions development workflow.
The LHCb software stack is developed in C++ and uses the Gaudi framework for event processing and DD4hep for the detector description. Numerical computations are done either directly in the C++ code ...or by an evaluator used to process the expressions embedded in the XML describing the detector geometry. The current system relies on conventions for the physical units used (identical as what is used in the Geant4 simulation framework) and it is up to the developers to ensure that the correct factors are applied to the values entered. Physical units are not primary entities in the framework, it is therefore not possible to check the dimensional consistency of the computation performed. In this paper we investigate the possibilities to add physical units and dimensions to the existing evaluator or to replace it by a more suitable system, and how this would integrate with the possible tools to express units in C++ code (such as boost::units).
LHCb software runs in very different computing environments: the trigger farm at CERN, on the LHC Computing Grid (LCG), on shared clusters or on software developer’s desktops. . . The old model ...assumes the availability of CVMFS and relies on custom scripts (a.k.a LbScripts) to configure the environment to build and run the software. It lacks flexibility and does not allow, for example running in container and it is very difficult to extend them to configure and run on new environments. This paper describes the steps taken to modularize those tools to allow for easier development and deployment (as standard Python packages), but also added integration with container technology to better support non standard environments.
Since 2015, with the restart of the LHC for its second run of data taking, the LHCb experiment has been empowered with a dedicated computing model to select and analyse calibration samples to measure ...the performance of the particle identification (PID) detectors and algorithms. The novel technique was developed within the framework of the innovative trigger model of the LHCb experiment, which relies on online event reconstruction for most of the datasets, reserving offline reconstruction to special physics cases. The strategy to select and process the calibration samples, which includes a dedicated data-processing scheme combining online and offline reconstruction, is discussed. The use of the calibration samples to measure the detector PID performance, and the efficiency of PID requirements across a large range of decay channels, is described. Applications of the calibration samples in data-quality monitoring and validation procedures are also detailed.
The physics software stack of LHCb is based on Gaudi and is comprised of about 20 interdependent projects, managed across multiple GitLab repositories. At present, the continuous integration (CI) ...system used for regular building and testing of this software is implemented using Jenkins and runs on a cluster of about 300 cores.
LHCb CI pipelines are python-based and relatively modern with some degree of modularity, i.e. the separation of test jobs from build jobs. However, these still suffer from obsoleted design choices that prevent improvements to scalability and reporting. In particular, the resource use and speed have not been thoroughly optimized due to the predominant use of the system for nightly builds, where a feedback time of 8 hours is acceptable. We describe recent work on speeding up pipelines by aggressively splitting and parallelizing checkout, build and test jobs and caching their artifacts. The current state of automatic code quality integration, such as coverage reports, is shown.
This paper presents how feedback time from change (merge request) submission to build and test reports is reduced from “next day” to a few hours by dedicated on-demand pipelines. Custom GitLab integration allows easy triggering of pipelines, including linked changes to multiple projects, and provides immediate feedback as soon as ready. Reporting includes a comparison to tests on a unique stable reference build, dynamically chosen for every set of changes under testing. This work enables isolated testing of changes that integrates well into the development workflow, leaving nightly testing primarily for integration tests.
Software is an essential and rapidly evolving component of modern high energy physics research. The ability to be agile and take advantage of new and updated packages from the wider data science ...community is allowing physicists to efficiently utilise the data available to them. However, these packages often introduce complex dependency chains and evolve rapidly introducing specific, and sometimes conflicting, version requirements which can make managing environments challenging. Additionally, there is a need to replicate old environments when generating simulated data and to utilise pre-existing datasets. Nix is a “purely functional package manager” which allows for software to be built and distributed with fully specified dependencies, making packages independent from those available on the host. Builds are reproducible and multiple versions/configurations of each package can coexist with the build configuration of each perfectly preserved. Here we will give an overview of Nix followed by the work that has been done to use Nix in LHCb and the advantages and challenges that this brings.
A continuous integration system is crucial to maintain the quality of the 6 millions lines of C++ and Python source code of the LHCb software in order to ensure consistent builds of the software as ...well as to run the unit and integration tests. Jenkins automation server is used for this purpose. It builds and tests around 100 configurations and produces in the order of 1500 built artifacts per day which are installed on the CVMFS file system or potentially on the developers’ machines.
Faced with a large and growing number of configurations built every day, and in order to ease inter-operation between the continuous integration system and the developers, we decided to put in place a flexible messaging system. As soon as the built artifacts have been produced, the distributed system allows their deployment based on the priority of the configurations.
We will describe the architecture of the new system, which is based on RabbitMQ messaging system (and the pika Python client library), and uses priority queues to start the LHCb software integration tests and to drive the installation of the nightly builds on the CVMFS file system. We will also show how the introduction of an event based system can help with the communication of results to developers.
LHCb is undergoing major changes in its data selection and processing chain for the upcoming LHC Run 3 starting in 2021. With this in sight several initiatives have been launched to optimise the ...software stack. This contribution discusses porting the LHCb Stack from x86_64 architecture to both architectures aarch64 and ppc64le with the goal to evaluate the performance and the cost of the computing infrastructure for the High Level Trigger (HLT). This requires porting a stack with more than five million lines of code and finding working versions of external libraries provided by LCG. Across all software packages the biggest challenge is the growing use of vectorisation - as many vectorisation libraries are specialised on x86 architecture and do not have any support for other architectures. In spite of these challenges we have successfully ported the LHCb High Level Trigger code to aarch64 and ppc64le. This contribution discusses the status and plans for the porting of the software as well as the LHCb approach for tackling code vectorisation in a platform independent way.
The LHCb experiment uses a custom made C++ detector and geometry description toolkit, integrated with the Gaudi framework, designed when the LHCb software was first implemented. With the LHCb upgrade ...scheduled for 2021, it is necessary for the experiment to review this choice and adapt to the evolution of software and computing (in terms of e.g multi-threading support or vectorization)
The Detector Description Toolkit for High Energy Physics (DD4hep) is a good candidate for the replacement for LHCb’s geometry description framework: it is possible to integrate it with the LHCb core software framework and its features theoretically match the requirements: in terms of geometry and detector description but also concerning the possibility to add detector alignment parameters and the integration with simulation tools.
In this paper we report on detailed studies undertaken to compare the feature set proposed by the DD4hep toolkit, to what is needed by LHCb. We show not only how the main description could be migrated, but also how to integrate the LHCb real-time alignment tools in this toolkit, in order to identify the main obstacles to the migration of the experiment to DD4hep.
We evaluate key patterns and estimate throughput bounds of simulated transformation of conventional high energy physics (HEP) data processing workflows to heterogeneous equivalents. The simulation ...parameter space includes the number of offloaded tasks, CPU/accelerator ratios of intra-task computations, offload latencies, and run time efficiency of offloaded computations. The simulation is performed for a diverse set of state-of-the-art event reconstruction scenarios from ATLAS, LHCb and CMS - the frontier HEP experiments of the Large Hadron Collider project.
PDF
