The integrated low-level trigger and data acquisition (TDAQ) system of the NA62 experiment at CERN is described. The requirements of a large and fast data reduction in a high-rate environment for a ...medium-scale, distributed ensemble of many different sub-detectors led to the concept of a fully digital integrated system with good scaling capabilities. The NA62 TDAQ system is rather unique in allowing full flexibility on this scale, allowing in principle any information available from the detector to be used for triggering. The design concept, implementation and performances from the first years of running are illustrated.
The use of GPUs to implement general purpose computational tasks, known as GPGPU since fifteen years ago, has reached maturity. Applications take advantage of the parallel architectures of these ...devices in many different domains. Over the last few years several works have demonstrated the effectiveness of the integration of GPU-based systems in the high level trigger of various HEP experiments. On the other hand, the use of GPUs in the DAQ and low level trigger systems, characterized by stringent real-time constraints, poses several challenges. In order to achieve such a goal we devised NaNet, a FPGA-based PCI-Express Network Interface Card design capable of direct (zero-copy) data transferring with CPU and GPU (GPUDirect) while online processing incoming and outgoing data streams. The board provides as well support for multiple link technologies (1/10/40GbE and custom ones). The validity of our approach has been tested in the context of the NA62 CERN experiment, harvesting the computing power of last generation NVIDIA Pascal GPUs and of the FPGA hosted by NaNet to build in real-time refined physics-related primitives for the RICH detector (i.e. the Cerenkov rings parameters) that enable the building of more stringent conditions for data selection in the low level trigger.
Graphical processors for HEP trigger systems Ammendola, R.; Biagioni, A.; Chiozzi, S. ...
Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment,
02/2017, Volume:
845
Journal Article
Peer reviewed
Open access
General-purpose computing on GPUs is emerging as a new paradigm in several fields of science, although so far applications have been tailored to employ GPUs as accelerators in offline computations. ...With the steady decrease of GPU latencies and the increase in link and memory throughputs, time is ripe for real-time applications using GPUs in high-energy physics data acquisition and trigger systems. We will discuss the use of online parallel computing on GPUs for synchronous low level trigger systems, focusing on tests performed on the trigger of the CERN NA62 experiment. Latencies of all components need analysing, networking being the most critical. To keep it under control, we envisioned NaNet, an FPGA-based PCIe Network Interface Card (NIC) enabling GPUDirect connection. Moreover, we discuss how specific trigger algorithms can be parallelised and thus benefit from a GPU implementation, in terms of increased execution speed. Such improvements are particularly relevant for the foreseen LHC luminosity upgrade where highly selective algorithms will be crucial to maintain sustainable trigger rates with very high pileup.
Graphics Processing Units for HEP trigger systems Ammendola, R.; Bauce, M.; Biagioni, A. ...
Nuclear instruments & methods in physics research. Section A, Accelerators, spectrometers, detectors and associated equipment,
07/2016, Volume:
824
Journal Article
Peer reviewed
Open access
General-purpose computing on GPUs (Graphics Processing Units) is emerging as a new paradigm in several fields of science, although so far applications have been tailored to the specific strengths of ...such devices as accelerator in offline computation. With the steady reduction of GPU latencies, and the increase in link and memory throughput, the use of such devices for real-time applications in high-energy physics data acquisition and trigger systems is becoming ripe. We will discuss the use of online parallel computing on GPU for synchronous low level trigger, focusing on CERN NA62 experiment trigger system. The use of GPU in higher level trigger system is also briefly considered.
A commercial Graphics Processing Unit (GPU) is used to build a fast Level 0 (L0) trigger system tested parasitically with the TDAQ (Trigger and Data Acquisition systems) of the NA62 experiment at ...CERN. In particular, the parallel computing power of the GPU is exploited to perform real-time fitting in the Ring Imaging CHerenkov (RICH) detector. Direct GPU communication using a FPGA-based board has been used to reduce the data transmission latency. The performance of the system for multi-ring reconstrunction obtained during the NA62 physics run will be presented.
NaNet is a modular design of a family of FPGA-based PCIe Network Interface Cards specialized for low-latency real-time operations. NaNet features a Network Interface module that implements RDMA-style ...communications both with the host (CPU) and the GPU accelerators memories (GPUDirect P2P/RDMA) relying on the services of a high performance PCIe Gen3 x8 core. NaNet I/O Interface is highly flexible and is designed for low and predictable communication latency: a dedicated stage manages the network stack protocol in the FPGA logic offloading the host operating system from this task and thus eliminating the associated process jitter effects. Between the two aforementioned modules, stand the data processing and switch modules: the first implements application-dependent processing on streams - e.g. performing compression algorithms - while the second routes data streams between the I/O channels and the Network Interface module. This general architecture has been specialized up to now into three configurations, namely NaNet-1, NaNet 3 and NaNet-10 in order to meet the requirements of different experimental setups: NaNet-1 features a GbE channel plus three custom 34 Gbps serial channels and is implemented on the Altera Stratix IV FPGA Development Kit; NaNet 3 is implemented on the Terasic DE5-NET Stratix V FPGA development board and supports four custom 2.5 Gbps deterministic latency optical channels; NaNet-10 features four 10GbE SFP+ ports and is also implemented on the Terasic DE5-NET board. We will provide performance results for the three NaNet implementations and describe their usage in the CERN NA62 and KM3NeT-IT underwater neutrino telescope experiments, showing that the architecture is very flexible and yet capable of matching the requirements of low-latency real-time applications with intensive I/O tasks involving the CPU and/or the GPU accelerators.
The KM3NeT-Italia underwater neutrino detection unit, the tower, consists of 14 floors. Each floor supports 6 Optical Modules containing front-end electronics needed to digitize the PMT signal, ...format and transmit the data and 2 hydrophones that reconstruct in real-time the position of Optical Modules, for a maximum tower throughput of more than 600 MB/s. All floor data are collected by the Floor Control Module (FCM) board and transmitted by optical bidirectional virtual point-to-point connections to the on-shore laboratory, each FCM needing an on-shore counterpart as communication endpoint. In this contribution we present NaNet3, an on-shore readout board based on Altera Stratix V GX FPGA able to manage multiple FCM data channels with a capability of 800 Mbps each. The design is a NaNet customization for the KM3NeT-Italia experiment, adding support in its I/O interface for a synchronous link protocol with deterministic latency at physical level and for a Time Division Multiplexing protocol at data level.
PDF
