+Model
ARTICLE IN PRESS
FUSION 4274 1–5
Available online at www.sciencedirect.com
Fusion Engineering and Design xxx (2007) xxx–xxx
ATCA data acquisition system for gamma-ray spectrometry
3
5
OF
R.C. Pereira a,∗ , J. Sousa a , A.M. Fernandes a , F. Patr´ıcio a , B. Carvalho a , A. Neto a , C.A.F. Varandas a , G. Gorini b , M. Tardocchi b , D. Gin c , A. Shevelev c
4
a
Associa¸ca˜ o EURATOM/IST Centro de Fus˜ao Nuclear, Instituto Superior T´ecnico, Lisboa, Portugal b Istituto di Fisica del Plasma, EURATOM-ENEA-CNR Association, Milan, Italy c A.F.Ioffe Physico-Technical Institute of the Russian Academy of Sciences, St. Petersburg 194021, Russian Federation
6 7 8
RO
9
Abstract
10
24
The gamma-ray spectrometer JET EP2 (Joint European Torus enhancement project 2) project aims to perform high-resolution gamma spectroscopy at very high count rate (up to few MHz). Traditional analogue electronic has count rate and pulse processing limitations (long dead-time, pile-up challenge). Digital pulse processing (DPP) systems have been shown to have better performance than analogue ones for processing neutrons or/and gamma-ray signals. DPP can synthesize almost any pulse response shape without the signal degradation associated to complex analogue paths. High-speed transient recorders (TR) with auto-trigger functionality are used to digitize and store the detailed shape of pulses. The data acquisition (DAQ) system provides sophisticated analysis/data reduction based on real time algorithms, implemented in field programmable gate arrays (FPGA), for Pulse Height Analysis (PHA) while resolving pulse pile-up of digitized pulses. This paper describes a new DAQ system for real-time pulse analysis. The system is based on the Advanced Telecommunications Computing ArchitectureTM (ATCATM ) and contains an ix86-based processor blade with up to 40 GFLOPS and a TR module interconnected through PCI Express (PCIe) links. TR module features: (i) 8 channels of 13 bit resolution with accuracy equal or higher than 11 bit to cope with the expected signal-to-noise ratio (SNR) of the input pulses; (ii) up to 500 MSamples/s sampling rate with the possibility to achieve 1 GSamples/s; and (iii) 2 or 4 GB of local memory. The core of the TR module is two FPGAs able to perform real-time processing algorithms such as PHA and pile-up resolution. This will allow data reduction by a factor of at least 6 and eventually spectra output in real-time. © 2007 Elsevier B.V. All rights reserved.
25
Keywords: Data acquisition; Gamma-ray spectroscopy; PHA; Pulse processing
11 12 13
DP
14 15 16 17 18 19 20
22 23
EC
TE
21
26
1. Introduction
1
The gamma-ray spectrometry project recently financed within the JET EP2 enhancement program is an upgrade of the existing gamma-ray spectrometers at JET. The project aims to perform high-resolution gamma spectroscopy at very high count rate up to few MHz. The analysis of gamma-ray energy spectra measured with collimated spectrometers allows the identification of different fast ions (H, D, T, 3 HE, 4 HE) while estimating their effective tail temperatures and relative concentrations [1]. Presently at JET there are three gamma spectrometers in use based on bismuth germanate (BGO) and NaI scintillator detectors. Two of them are installed at the roof lab and the other in a tangential view. Although functional they present limitations
RR
2 3 4 5 6
8 9 10 11 12 13 14
Q1
1 2
∗
UN
CO
7
Corresponding author. Tel.: +351 239 410108; fax: +351 239 829158. E-mail address:
[email protected] (R.C. Pereira).
in count rate, energy resolution and large neutron background. The new spectrometer diagnostics will be located on the roof lab replacing the two existing gamma spectrometers. Three new gamma spectrometers with complementary performance, will be installed one with a high energy resolution HPGe and two of high efficiency, high rate with large detection crystal, one made from LABr3 scintillator and one from LYSO scintillator. The key figure of merit of this enhancement is the count rate capability which shall exceed 0.5 MHz before pileup and gain drifts take place affecting detector response. These effects will be controlled using a suitable data acquisition system in order to face the expected total throughput of 2 MHz of total count rate from which a variable fraction are useful gamma events [2]. To cope with the above-mentioned high count rate and high-resolution spectrometers a new DAQ system with DPP techniques for real-time PHA with resolved pile-up was developed. DAQ system features: (i) amplitude resolution of 13 bits to cope with the expected SNR of the input pulses, (ii) sampling
0920-3796/$ – see front matter © 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.fusengdes.2007.10.011
Please cite this article in press as: R.C. Pereira, et al., ATCA data acquisition system for gamma-ray spectrometry, Fusion Eng Des (2007), doi:10.1016/j.fusengdes.2007.10.011
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
+Model
ARTICLE IN PRESS
FUSION 4274 1–5
R.C. Pereira et al. / Fusion Engineering and Design xxx (2007) xxx–xxx
44
2. System description
42
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
The DAQ system of the new gamma ray spectrometer diagnostic is based on the ATCATM PICMG 3.0 standard [3]. The ATCATM base specifications define a board and architecture sharing a common backplane with interconnections based on a full mesh of serial gigabit communication links. Each slot is interconnected to all others through x1, x2 and x4 links with a maximum throughput capacity of 800 MB/s. each controlling a set of ATCA digitizer modules. The ATCA system is composed by: (i) 14 slot ATCA shelf (sub-rack); (ii) the processor blade, a low-cost ATX motherboard mounted on an ATCA carrier module, connected to the ATCA backplane through an x16 PCIe link (3.2 GB/s) [4], capable of processing the data from four ATCA cards; and (iii) one acquisition module with 8 free-running 13 bit channels with sampling rate of 250 MSPS. The unit is interfaced to the JET Control and Data Acquisition System (CODAS) through 100 Mbit/1 Gbit Ethernet port. A PCI Express to ATM interface will connect the system to the JET Real-time Network for real-time monitoring or control purposes. DAQ system will host a FireSignal node [5] providing an easy way of integration of this diagnostic at JET.
DP
41
TE
40
EC
39
RR
38
The paper scope is the ATCA TR module description, Fig. 1. The TR module has eight 250 MSPS free-running channels with 13 bit of resolution divided into two blocks. Each block with four channels directly connected to its respective FPGA and DDR2 memory. Although the TR module has eight channels, only five channels are needed for this application divided as follows: (i) two channels are interleaved leading to one 500 MSPS channel for the high efficiency, high rate spectrometer with large detection crystal from LABr3 scintillator; (ii) another two channels are interleaved leading to one 500 MSPS channel for the high efficiency, high rate spectrometer with large detection crystal from LYSO scintillator; (iii) one channel at 250 MSPS to acquire the high energy resolution HPGe spectrometer. The first block will have one 500 MSPS and one 250 MSPS channels and the other block only one 500 MSPS channel. At speeds above 200 MHz, only fast programmable devices can be used for data processing and transfer. With this approach two FPGAS, one per block, from XilinxTM Virtex4 family (XC4VFX60-1152) [6] are capable of: (i) controlling analogue inputs calibration; (ii) processing data; (iii) managing data storage; and (iv) providing a gigabit communication interface, 4 lane (4×) PCIe link (4× 2.5 GHz full duplex). The FPGA is also used for data reduction by doing PHA while resolving pile-up. Moreover it is also responsible for the complex managing modes of triggering (auto-triggering functionality) and serial peripheral interface controller. At the time TR module was designed, the fastest commercially available ADC was the 13-bit ADC rated to 250 MSPS, ADS5444 from Texas Instruments® (TI). Faster 13 bit ADCs can be achieved with the already mentioned interleaved architectures, although it represents an immediate threat to the 11-bit dynamic range performance (68 dB). A new pin-compatible
CO
36 37
2.1. Module architecture
UN
35
OF
43
rate of 250 MSPS capable of 1 GSPS for accurate pulse shaping, (iii) up to 4 GB of local memory (up to 2 GB per block) to store pulses at the expected rate during the experiment duration. The data acquisition system provides sophisticated analysis/data reduction based on real time algorithms, implemented in FPGA, for PHA and pulse pile-up resolution of digitized pulses. The following sections will present the TR module and how data is processed in order to allow a higher operation time by a factor of at least 6 comparing to raw pulse storage.
34
RO
2
Fig. 1. ATCA DAQ module block diagram.
Please cite this article in press as: R.C. Pereira, et al., ATCA data acquisition system for gamma-ray spectrometry, Fusion Eng Des (2007), doi:10.1016/j.fusengdes.2007.10.011
66
67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
+Model
ARTICLE IN PRESS
FUSION 4274 1–5
106
2.2. PCIe switch
102 103 104
123
2.3. PCIe endpoint
108 109 110 111 112 113 114 115 116 117 118 119 120 121
131
2.4. Operation mode and data storage
132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150
EC
129
RR
128
The FPGA is directly connected to the free-running ADC channels acting as temporary data buffer, real-time eventmanager, time stamping and performing some high speed algorithms like digital level trigger detection, PHA while resolving pile-up. TR module allows two operating modes: (i) raw data mode, which can be divided into two others: (a) all acquired data is stored in the DDR2 memory. This type of acquisition is for test purposes, because three channels sampling at 250 MSPS will fill up the 1 GB memory of block 1 in less than 1 s and the other memory block in 1 s; (b) only pulse data, if the pulse is defined with 20 samples (40 ns) it will store 48 bytes per pulse, where 8 bytes have the time stamp and channel information of the acquired pulse; (ii) processed data, meaning the PHA and the pile-up resolution were applied and the final data gathered in a word of 64-bits with the information of pulse energy, related channel and time of pulse occurrence. This reduces by a factor of 6 the memory’s filling time, allowing a longer operating time. For pulses larger than 40 ns the reduction is even better than the
CO
127
UN
126
TE
130
125
factor of 6. Processed data can be written to memory or directly sent to the controller module through PCIe links. Another mode that might be implemented in the future is spectra mode where a spectra is built in-real-time and sent through PCIe link. The three operation modes are depicted in Fig. 2. Time marks have the resolution of 4 ns and are given by a 48bit counter with a time span of T = 248 × 4 ns > 24 h, incremented synchronously with the data acquisition clock. 3. Post processing
This PCIe-based module is PCI compliant by means of a PCI compatible configuration that supports familiar transactions such as memory, input/output (IO) and configuration read/write transactions actions. The memory, IO and configuration address space model is the same as PCI address spaces allowing existing operating systems (OS) and driver software to run in PCI Express system without any modifications.
124
Fig. 2. DAQ module operation mode.
DP
122
ExpressLaneTM PEX 8516 devices provide PCIe switching capability, enabling users to add scalable, high-bandwidth, nonblocking interconnection to a wide variety of applications [7]. This device has a flexible port width configuration and is used as a fan-out application. The upstream and downstream ports are all configured as x4. This switch does not allow an upstream port with spread spectrum clocking (SSC) and a downstream port with constant frequency clock (internal oscillator). Consequently the 100 MHz PCIe reference must be supplied by the root complex (controller module connected to the upstream port). Virtex-4 PCIe endpoint must be synchronously clocked to the root complex and the clock input must be 250 MHz. The 100 MHz PCIe reference must be multiplied to 250 MHZ while at the same time remaining compliant to the jitter specifications required by the Virtex-4 Multigigabit Transceivers (MGTs).
107
RO
105
ADC rating 500 MSPS also from TI (ADS5463) can be used if the dynamic performance of the interleaved channels goes below the 10.5 effective bits of resolution. Each FPGA is in charge of a x4 PCIe connection. A PCIe switch is needed to allow the user to access both blocks of the module as data arrive to both block channels.
100 101
3
OF
R.C. Pereira et al. / Fusion Engineering and Design xxx (2007) xxx–xxx
Digital pulse processing can occur in both FPGA and the ix86 processor. The former excels performing high throughput parallel simple operations directly on the high-rate sampled data and the latter supports more complex algorithms at moderate rates. The processing at FPGA level has been theoretically tested and the algorithm is represented at Fig. 3. Each ADC is directly connected to the: (i) processing block; (ii) edge detector (ED) and (iii) circular buffer, where the acquired pulse is temporarily stored. At the processing block the acquired pulse d(n) goes through the first stage of the digital trapezoidal shaper (DTS), a high-pass deconvolver (HPD), this stage when supplied with a sampled exponential pulse with a decay time constant τ, returns a step signal that can be used to some extent to determine the pulse energy, when the distance between steps is short, the quadratic component in the HPD output, due to drifts in the input signal, is minimized, making it acceptable for peaks with few points, the risk of overflow is avoided due to the use of two’scomplement arithmetic. The output signal from the HPD, h(n), goes on through the rest of the DTS, consisting of two delay subtract modules (DS). These modules DS1 and DS2 require one parameter each, k and l, respectively. The output of the DTS, is a trapezoid with a rising (falling) edge with a duration of the minimum between k and l and a flat top with a duration of the absolute value of the difference between k and l [8]. Every time ED detects a pulse it will trigger the energy resolvers, ER1 or ER2 , calculating the number of points between pulses, if the number is greater or equal than k + 2 × l + 2 (min-
Please cite this article in press as: R.C. Pereira, et al., ATCA data acquisition system for gamma-ray spectrometry, Fusion Eng Des (2007), doi:10.1016/j.fusengdes.2007.10.011
151 152 153 154 155 156 157 158 159
160
161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189
+Model
ARTICLE IN PRESS
FUSION 4274 1–5
R.C. Pereira et al. / Fusion Engineering and Design xxx (2007) xxx–xxx
RO
OF
4
Fig. 3. Data processing algorithm block diagram.
DP
TE
195
EC
194
RR
193
To prove the feasibility of the just described algorithm, pulses with 20 ns of decay time were generated with periodic noise, 75% of pile-up events and sampled at 500 MHz. Expecting so many pile-up events, a compromise was reached when using the DTS, defining k = 2 and l = 8 (instead of k = 2 and l = 30, for example), so ER1 could be used in a greater number of pulses, not just the ones at least 3 decay times apart (95% pulse area),
CO
192
imum points so that ER1 can operate) then, ER1 determines the energy, otherwise, ER2 determines the energy that after being rescaled can be added to the final energy spectrum. ER1 settles the energy from the flat top values and the estimated baseline of the trapezoids. ER2 settles the energy calculating the difference between steps after compensating for the slope due to the offset in the input signal.
UN
190 191
Fig. 4. Spectra obtained from the simulation of 1E5 pulses with two energy amplitudes with Gaussian noise, base line shift, and drift and 75% of pile-up events: (a) peak pulse discriminator (PPD), pulse energies measured directly from the sampled data by capturing the maximum and subtracting the baseline; (b) HPD, energy of peaks processed by ER2; (c) DTS, energy of the remaining peaks, processed by ER1; (d) HPD + DTS, represents the sum of the p1 (n) with p2 (n) outputs.
Please cite this article in press as: R.C. Pereira, et al., ATCA data acquisition system for gamma-ray spectrometry, Fusion Eng Des (2007), doi:10.1016/j.fusengdes.2007.10.011
196 197 198 199 200 201 202
+Model
ARTICLE IN PRESS
FUSION 4274 1–5
R.C. Pereira et al. / Fusion Engineering and Design xxx (2007) xxx–xxx 203 204 205 206 207 208 209 210 211 212 213
this slightly increased the error of the 25% isolated pulses, but greatly reducing the error in 52% of the piled-up pulses. With these parameters, ER1 requires at least 18 samples between pulses (10 from the trapezoid and 8 extra for the baseline estimate). The spectra built with the processed data are the sum of ER1 output and the ER2 output. This way using the DTS for 77% of the pulses and the HPD for the remaining 23%, we lose only 3%, which are pulses that are so close that are perceived as one, in the spectrum showing as a peak with their energies summed. The following results are depicted at Fig. 4.
5
Acknowledgements This work has been carried out in the frame of the Contract of Association between the European Atomic Energy Community and Instituto Superior T´ecnico (IST) and of the Contract of Associated Laboratory between Fundac¸a˜ o para a Ciˆencia e Tecnologia (FCT) and IST. The content of the publication is the sole responsibility of the authors and it does not necessarily represent the views of the Commission of the European Union or FCT or their services.
221 222 223 224 225
RO
220
DP
219
TE
218
EC
217
RR
216
The peak broadening is due to the fact that: (i) HPD ER2 works with a minimum of two points between pulses and (ii) one or zero points distance pulses are seen as just one pulse and appear with their energies summed up in the energy spectrum. As expected HPD spectra presents worse broadening. As pulse pile-up is unavoidable on high count rate detectors, spaced events can be obtained with fast response detectors. The new LYSO and LABR3 scintillation detectors with short decay-time are expected to improve the spectra peaks broadening. If in the future a more accurate pulse shaping is essential the TR module is prepared to interleave four channels achieving 1 GSPS of sampling rate.
[1] V.G. Kiptily, et al., ␥-ray diagnostics of energetic ions in JET, Nuclear Fusion Q2 42 (2002) 999–1007. [2] JET-EP2 - Project Plan: “Gamma-ray spectrometry-GRS”, 2006, JET internal document. This document can be requested to
[email protected]. [3] AdvancedTCA® , PICMG® 3.0 Revision 2.0, AdvancedTCA® Base Specification, March 18, 2005. [4] AdvancedTCA® , PCI ExpressTM /Advanced Switching for AdvancedTCA® Systems, May 21, 2003. [5] A. Neto et al., The control and data acquisition software for the gamma-ray spectroscopy ATCA sub-systems of JET-EP2 enhancements, presented at 6th IAEA TM on Control, Data Acquisition, and Remote Participation for Fusion Research, Inuyama, Japan. [6] http://www.xilinx.com/. [7] http://www.plxtech.com/. [8] Digital techniques for real-time pulse shaping in radiation measurements, Nuclear Instrum. Meth. Phys. Res. A 353 (1994) 261–264.
CO
215
4. Final consideration
UN
214
OF
References
Please cite this article in press as: R.C. Pereira, et al., ATCA data acquisition system for gamma-ray spectrometry, Fusion Eng Des (2007), doi:10.1016/j.fusengdes.2007.10.011
226
227 228 229 230 231 232 233 234
235
236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251
