UNIVERSITY OF APPLIED SCIENCES WILDAU

Faculty of Engineering / Industrial Engineering

cooperative course Telematics

INTERNSHIP REPORT

Development of a remote configuration tool for Xilinx

Virtex FPGAs used in CERN LHC

Report handed in at:

by:

26. August 2008

Stefan Kramer T-06

supervising professor:

Prof. Dr. A. Fabig

supervisor during the internship:

Prof. H. Helstrup

Contents
List of Figures

List of Tables

Glossary

1 Abstract

2 Preamble
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Description of the task . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7
7
7

3 ALICE
3.1 General description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Time projection chamber . . . . . . . . . . . . . . . . . . . . . . . . . . .

8
8
10

4 Detector electronics
4.1 Detector-Control-System . . .
4.2 Front-End-Electronics Server .
4.3 Readout-Control-Unit . . . .
4.4 Reconfiguration of the FPGA

.
.
.
.

11
11
11
12
13

.
.
.
.
.

14
14
15
15
16
18

.
.
.
.

5 Remote Configuration Tool

5.1 Brief introduction . . . . . . . .
5.2 Generating configuration files .
5.2.1 Frame Files . . . . . . .
5.2.2 Binary-Command-Blocks
5.3 SSH Communication . . . . . .

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

.
.
.
.

.
.
.
.
.

6 Conclusion

19

Bibliography

20

List of Figures
3.1
3.2

general layout of ALICE . . . . . . . . . . . . . . . . . . . . . . . . . . .

drifting of the particles in the TPC . . . . . . . . . . . . . . . . . . . . .

8
10

4.1

functional sketch of the RCU

. . . . . . . . . . . . . . . . . . . . . . . .

12

5.1
5.2

Front End of the configuration tool . . . . . . . . . . . . . . . . . . . . .

SSH-Upload window . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14
18

List of Tables
5.1
5.2
5.3

Initial-Configuration-Block . . . . . . . . . . . . . . . . . . . . . . . . . .
Scrubbing-Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Frame-by-Frame-Readback-Block . . . . . . . . . . . . . . . . . . . . . .

16
17
17

Glossary
ALICE

A Large Ion Collidor Experiment

APR

Active Partial Reconfiguration

CERN

Conseil Europeen pour la Recherche Nucleaire (European Council for Nuclear

Research)

Co2

Carbon dioxide

DCS

Detector Control System

FEC

Front End Card

FEE

Front End Electronics

FPGA

Free Programmable Gate Array

LHC

Large Hadron Collidor

N2

Nitrogen

Ne

Neon

QGP

Quark Gluon Plasma

RCU

Readout Control Unit

TeV

Terra electron Volt

TPC

Time projection chamber

1 Abstract
Radiation effects may influence the behaviour of logic devices. So the FPGAs that are used
in the ALICE Detector-Control-System (DCS) may expect these errors. To correct them the
data has to be prepared. In this report a solution to generate data for reconfiguration of
FPGAs is described.

Zusammenfassung
Strahlung beeinflusst das Verhalten von elektronischen Bauteilen. Die FPGAs, welche im
Detector-Control-System (DCS) des ALICE Experiments zum Einsatz kommen, arbeiten in
einer solchen Umgebung. Um den Auswirkungen der Strahlung zu begegnen wird der Speicher der FPGAs neu beschrieben. Die dafur benotigten Daten mussen im Vorfeld aufbereitet
werden. In diesem Praktikumsbericht wird ein Werkzeug vorgestellt, mit dessen Hilfe man
diese Daten generieren kann.

2 Preamble
2.1 Introduction
In these days at CERN1 near Geneva in Switzerland the Large-Hadron-Collidor (LHC) is
commissioned for its first run on September 9th. When this machine is completed it will be
the biggest one ever built. Physicists hope to investigate with this machine the origin of our
todays matter. Therefore several experiments are running. One of them is ALICE2 , where the
origination of today known particles like protons and neutrons is investigated.

2.2 Description of the task

Several detectors of the ALICE experiment at CERN LHC contain FPGAs. Since these FPGAs
run in a radiation environment radiation is influencing the FPGAs memory. Using a feature
called Active partial reconfiguraion loosing the program in the memory can be avoided.
Therefore in the past the data was generated using a tool which runs directly on Detector
control system. Because this is pretty slow and usable only for one type of FPGA it has to
be replaced by a more comfortable and flexible version.

1
2

Conseil Europeen pour la Recherche Nucleaire (European Council for Nuclear Research)
A Large Ion Collidor Experiment

3 ALICE
3.1 General description
ALICE is part of the LHC which is currently commissioned at CERN in Geneva. LHC consists
of a 27 km circumference tunnel where hadrons (protons and neutrons from atomic nuclei)
are accelerated up to nearly speed of light by two beams of opposite directions. The hadrons
collide on four points where the rings intersect. At these points detectors are installed to
grab any kind of data concerning the collision. Figure 3.1 shows the general layout of ALICE.

Abbildung 3.1: general layout of ALICE

Source: http://aliceinfo.cern.ch

The acceleration to nearly speed of light gives the protons an energy of 7TeV, so that
when two protons collide the energy doubles up to 14 TeV. To imagine the energy, it can be

assumed, that the motion energy of a flying mosquito is about 1TeV, but the space in the
accelerator is many times smaller than a mosquito.
The goal of ALICE is to investigate the situation immediately after the big-bang. This
state is also known as QGP (Quark-Gluon-Plasma) and has last for only 10 microseconds
after the big bang. In this state, the particles and structures as we know them today, have
their origin. They were established when the QGP cooled down and collapsed to protons and
neutrons.
To reconstruct this state high energy is necessary, so that the particles can melt and squeeze.
This enormous energy is reached by acceleration to nearly speed of light. For more information
about ALICE see [1].

3.2 Time projection chamber

The Time projection chamber (TPC), located in the middle of Figure 3.1, is the main part
of ALICE. It consists of a 88 m3 cylinder filled with gas (Ne/Co2 /N2 ). On both ends of the
cylinder planes can be found, which are divided in 18 equal sectors. Each sector contains 121
Front-End-Cards(FEC), so there are 4356 FECs in the whole TPC.
The main task of TPC is to provide measurement for particle momentum and particle identification. To get this, the lead particles ionize the gas and then drift towards the detector
plates, which are located in each sector of TPC. The following figure illustrates the drifting
of the particles.

Abbildung 3.2: drifting of the particles in the TPC

The FECs of each sector are connected to a Readout-Control-Unit (RCU), which grabs
the data from the FECs and sends them to the Data-Acquisition-System where either long
time saving for later offline reconstruction is made or online High-Level-Triggering. During
the experiment data rates up 80 MByte per event occur. When ALICE is running the events
are sampled by a rate of 100 kHz, so that the data rate increases up to hundreds of GByte.

10

4 Detector electronics
4.1 Detector-Control-System
The Detector-Control-System (DCS) used in many different detectors of ALICE. In the TPC
it is mounted on the RCU card and provides remote access to their functions. This board
contains an ALTERA FPGA which comprises itself an ARM9 controller. In addition to this
there are an 8 MB Flash ROM (radiation tolerant) and 32 MB SDRAM, an ethernet interface
and some other auxiliary devices mounted. This basic equipment allows running a light-weight
version of Linux, so that the devices can be accessed from remote.

4.2 Front-End-Electronics Server

One main application that runs on the DCS board is the Front-End-Electronics Server(FeeServer)[6].
This program monitors data from the underlying FECs. It is also able to receive user commands to control this underlying hardware. One task of the FeeServer is to prepare the
reconfiguration of the Xilinx FPGA (see section 4.4).

11

4.3 Readout-Control-Unit
The TPC Readout-Control-Unit (RCU) was developed to readout, control and monitor any
kind of FECs in the TPC. The physical layout consists of a main FPGA which controls,
monitors, configures and reads out the FECs. The functional design can be distinguished in
four blocks:
ALTRO3 -Bus controller is used to connect the FECs to the RCU [3].
monitoring and safety module, that monitors power state, voltages, currents and temperature state of the FECs
interface to Detector Data Link Source Interface Unit (DDL SIU), which provides a
protocol to send data to the Data Acquisition (DAQ) or receives configuration data
for the FECs
interface to the DCS / Trigger board, this receives trigger data or configuration data
for the FECs or transmits them to the DCS
In the following figure you can see the functional sketch of the RCU (Figure 4.1).

Abbildung 4.1: functional sketch of the RCU

Source: http://web.ift.uib.no/ kjeks/wiki

ALICE TPC Readout

12

4.4 Reconfiguration of the FPGA

Since the FPGAs are used in a radiation environment, Single-Event Upsets (SEUs) can occur.
SEUs are single bit flips that can change the content of any kind of static memory element
e.g. RAM cells or flip-flops [5]. A SRAM based FPGA is the best solution because of strong
real-time conditions during the experiment, that flash based FPGAs do not fulfill.
To face the effects of SEUs there are two different ways:
correction of a single frame - Reading the frame from the FPGA and replacing it
with a copy of the original one. The big advantage of this method is, that only one
single frame has to be set to write mode, so only a very small part of the logic
is affected. The disadvantage is, that there has to be an extra logic which contains
algorithms to do the reconfiguration and an extra memory which holds the data. Of
course this part has to be radiation tolerant.
scrubbing of the memory - This operation will reload not only a single frame, but a
certain amount of frames where the SEU is promised. Using this method would produce
a much greater downtimeof the whole logic.
In case of the RCU Xilinx Virtex-II Pro FPGA both method are used to reconfigure.
Therefore another FPGA (Actel APA075 - flash-based) and an external flash memory are
used. There are three different types of communication between the DCS and the RCU
board: normal operation, SelectMAP mode and flash mode. To write the original data to
the flash memory the flash mode is used, where the DCS board has direct access to the
flash memory. It is also possible to access the memory of the Xilinx FPGA direct from the
DCS card. This is done in SelectMAP mode. In this mode the DCS card has full access to
the SelectMAP interface (8 bit bidirectional data bus to access the configuration logic) of
the Xilinx. Last but not least there is the normal operation mode where the Actel device is
doing the Active-Partial-Reconfiguration (APR) of the Xilinx. For scrubbing the whole device
there are three different possibilities: sending a scrubbing command to the DCS card and do
scrubbing once, scrub for a given number of repetitions or scrub until an abort command is
received. The other way is to read back each frame and compare it to the original data stored
in the flash memory. To do this configuration a previously generated bit file is necessary. For
the first method the bit file can be used as whole, for the second is has to split up to the
number of frames used for reconfiguration of the according Xilinx device, e.g. the Virtex-II
Pro has 1320 frames as whole, but only 936 are used for APR, the other frames are BRAM
frames which contain data that should not be overwritten.

13

5 Remote Configuration Tool

5.1 Brief introduction

Abbildung 5.1: Front End of the configuration tool

This tool enables the user one the one hand to split a configuration file up to several frame
files and on the other hand to generate binary command blocks. To increase the usability the
program provides an additional SSH part where the user can either load the generated files
up to a dedicated server or program the devices on the RCU directly.
Since this tool should be able to handle different versions of Virtex FPGAs an external
configuration file is used. This configuration file is XML based and provides information for
all types of FPGAs. Furthermore all information concerning the binary command blocks are
included.

14

5.2 Generating configuration files

5.2.1 Frame Files
Generating the frame files from the original bit file is done in the following steps:
1. read all configuration data from the XML file
This means searching for information to the Virtex device specified in the GUI. If the
XML file contains data for that specific device, it is read out. The data are the total
number of frames, the number of frames for each type(IOB, IOI, CLB, BRAM, BRAMIC
and GCLK frames), the number of columns for each frame type and the frame length
(the same for each frame type).
2. read the original bit file and find the synchronization word
All data that comes before that 32 bit word is information that is not necessary for
reconfiguration.
3. find the device identifier and compare it to the one found in the XML file
This is done to ensure that the data from the XML file and the bit file belong to the
same device type.
4. find end of header word
If the bit file does not contain this word, there is no sign available that marks the
beginning of the data block.
5. read the rest of the file
The rest of the bit file is buffered to avoid multiple read/write operations that would
decrease the performance.
6. generate the frames
Due to the structure of the FPGAs memory, the name of the frame files get a special
scheme, which looks like this: frame$block.$major.$minor.hex, e.g. frame0.19.12.hex.
The block number is either 0 or 2, because block 1 indicates the BRAM block, which
contains data and should not be overwritten. The major number is assigned new for
each block and is calculated depending on the different frame types. The minor number
is assigned to each new major number continuously.
Depending on the frame length (e.g. the Virtex-II frames have a length of 424 byte)
the data is read out of the buffer and then written to a file.

15

This algorithm is used to generate raw frame files, read frame files, write frame files and
the binary command for frame-by-frame readback (see section 5.2.2). The difference between
raw frames and read/write frames is that these frames contain header and footer fields where
information about the addresses the data is stored to and some commands is provided. Since
these values are device specific they are also available in the XML configuration file.

5.2.2 Binary-Command-Blocks
Besides generating frame files for APR the program is also able to generate blocks of binary
commands. These command blocks are used in the FeeServer (introduced briefly in section
4.2). As explained in [7] the FeeServers ControlEngine (CE) contains classes that represent
the hardware devices the DCS and RCU cards contain. One of these devices is the Actel
FPGA which is responsible for the reconfiguration of the Xilinx device. To handle this reconfiguration procedure the Actel needs instructions and data. This both is contained in the
binary command blocks ([7]) generated by the tool developed during this internship.
There are three kinds of command blocks being generated.
1. Initial-Configuration-Block
Header
0xF4060000

Initial addresses
...

Data
initial bitfile

Command tailer
0xDD330000

Tabelle 5.1: Initial-Configuration-Block

When programming a new firmware version to the Xilinx also the copy in the flash
memory has to be updated. To do this the Initial-Configuration-Block as seen above is
used. The header field contains the command what to do with the following data. The
fields below are addresses in the flash memory and the configuration file for the Xilinx
device. Closing this command block at the end the command tailer can be found. This
field consists of a command end marker and the command version.

16

2. Scrubbing-Block
Header
0xF4070000

Scrubbing addresses
...

Data
scrubbing bitfile

Command tailer
0xDD330000

Tabelle 5.2: Scrubbing-Block

In case of scrubbing, the block contains another bitfile which is surrounded by commands and addresses sent to the Actel. Similar to the Initial-Configuration-Block this
block also contains a command which tells the Actel what to do with the data, some
addresses for the flash memory and the data itself.
3. Frame-by-Frame-Readback-Block
Header
0xF40303A8

Frame
dresses
...

ad-

Data
control
information
concerning the
frames

Readframe
Data
readframes

Writeframe
Data
writeframes

Command tailer
0xDD330000

Tabelle 5.3: Frame-by-Frame-Readback-Block

The frame-by-frame-readback command is the most complex. It contains addresses
where to write the frames in the flash and also 3 frame info registers (composed in
the Data block) that were introduced to gain forward compatibility[8].

17

5.3 SSH Communication

To increase usability and workflow the program enables the user to load the generated files
up to the DCS card. Since on the DCS card runs an embedded Linux there is also a SSH
server available. This means a user can log on to the card using any kind of SSH-tool like
ssh or putty. This way is quite intricately and so the SSH section (using the free SSH library
libSSH4 ) provides similar functionality. Using this, hostname, username and password must
be known. If this was all correct, the upload window becomes visible.

Abbildung 5.2: SSH-Upload window

In this dialogue the user can choose either a file or a folder to load up to a dedicated
location on the server the user has write access to. To ensure a safe communication
the SFTP subsystem is used. Establishing a SFTP session a valid SSH session must
exist. File operations on this system are quite similar to normal POSIX5 file operations.
destFile = sftp open(sftpSession, destPath, O CREAT | O TRUNC | O WRONLY, NULL);
//destFile can be compared to a normal POSIX file descriptor
The only difference is that the SFTP session has to be added. Specifying the path where the file has to be opened and the opening flags (creating the file, truncating the content if
the file exists and opening the file for write access only) are similar to the POSIX standard.

4
5

http://0xbadc0de.be/wiki/libssh:libssh
Portable Operating System Interface, http://standards.ieee.org/regauth/posix

18

6 Conclusion
According to the introduction, the tool developed during this internship enables the user to
generate reconfiguration data from remote. This makes the work easier and the flexible design
ensures future extension and association to other FPGAs. Furthermore the ability to load the
generated files directly to system there are needed makes the work more efficient. However
it must be mentioned, that for a complete remote configuration the ability of programming
the Xilinx device from remote is not yet implemented completely. If this is working no direct
access to the DCS cards will be necessary.

19

Literaturverzeichnis
[1] The ALICE Collaboration, ALICE Technical Proposal for A Large Ion Collidor Experiment at
CERN LHC, Technical Report, CERN, 1995
[2] Gonzales Gutierez, C. et al., The ALICE TPC Readout Control Unit, in Nuclear Science Symposium Conference Record, vol. 1, pp. 575 - 579, 2005
[3] Esteve Bosch, R. et al., The ALTRO Chip: A 16 channel A/D and Digital Processor for Gas
Detectors, in IEEE Transactions on Nuclear Science, vol. 50 no. 6, pp. 2460 - 2469, 2003
[4] Richter, M. et al., The control system for the Front-End Electronics of the ALICE Time Projection Chamber, in IEEE Transactions on Nuclear Science, vol. 53 no.3, pp.980 - 985, 2006
[5] Carmichael, C. et al., Correcting Single-Event Upsets through Virtex Partial Reconfiguration,
Xilinx Application Note XAPP216, v1.0, 2000
[6] Bablok, S., Development and implementation of a safe and efficient communication software
in a heterogeneous system environment of a major research project, Diploma Thesis, University
of Applied Sciences Worms, 2004
[7] Fehlker, D., Development and commissioning of a software environment for controlling and reconfiguration of Xilinx Virtex FPGAs, Diploma Thesis, University of Applied Sciences Mittweida,
2007
[8] http://web.ift.uib.no/ kjeks/wiki/index.php?title=The Actel software device in the
FeeServer CE#Composition of the Binary command blocks, 2008-08-15

20


ERKLARUNG
Hiermit versichere ich, dass ich die vorliegende Arbeit selbststandig und ohne Benutzung

unzulassiger Hilfsmittel angefertigt habe. Wortliche oder sinngemae Ubernahmen

aus anderen Ver
offentlichungen sind mit genauer Angabe der Quelle kenntlich gemacht.

Bergen, den 15. August 2008

Stefan Kramer