Study of White Rabbit Based Real-Time Ethernet For Distributed Control System

STUDY OF WHITE RABBIT BASED REAL-TIME
ETHERNET FOR DISTRIBUTED CONTROL SYSTEM
An Internship-cum-Project Report
Submitted By:
SIDDH RAMAN PANT (190170111075)

AKHILESH BAHUGUNA (190170111011)
In partial fulfillment for the award of the degree of:

BACHELOR OF ENGINEERING
in
Electronics and Communication Engineering
Vishwakarma Government Engineering College, Chandkheda
Gujarat Technological University, Ahmedabad

May, 2023
Certificate
This is to certify that this project report entitled "Study of White Rabbit Based Real-
Time Ethernet for Distributed Control System" is the record of work carried out by
Siddh Raman Pant (bearing enrollment number 190170111075) under our supervision
and guidance towards the partial fulfilment in the subject of Internship / Project
(3181101) for the degree of Bachelor of Engineering in Electronics and Communication
during the 8th semester in the academic year 2022-2023 of Gujarat Technological
University, Ahmedabad.
The work submitted has, in our opinion, reached a level required for acceptance as per the
norms and quality mandated by the University. The results embodied in this internship
report, to the best of our knowledge, have not been submitted to any other university or
diploma.
Date: 10th May, 2023
Prof. Shreeji H. Shah Dr. Arun B. Nandurbarkar
Assistant Professor, EC Head of the Department, EC

No Plagiarism Declaration
We hereby declare that
1. We know that plagiarism means taking and using the ideas, writings, works, or
inventions of another as if they were one’s own. We know that plagiarism not
only includes verbatim copying, but also the extensive use of another person’s
ideas without proper acknowledgement (which includes the proper use of
quotation marks). We know that plagiarism covers this sort of use of material
found in textual sources and from the Internet.
2. We acknowledge and understand that plagiarism is wrong.
3. We understand that our research must be accurately referenced. We have
acknowledged all material and sources used in its preparation, whether they be
books, articles, reports, lecture notes, any other kind of document, and
electronic or personal communication.
4. This project report (Study of White Rabbit Based Real-Time Ethernet for
Distributed Control System) is the culmination of our own research work carried
out at IPR as a team. We acknowledge that copying someone else’s report, or part
of it, is wrong, and that submitting identical work to others constitutes a form of
plagiarism.
5. We have not allowed, nor will we in the future allow, anyone to copy our work
with the intention of passing it off as their own work.
Date: 10th May, 2023
Siddh Raman Pant Akhilesh Bahuguna

GTU PMMS Team ID: 318853 GTU PMMS Team ID: 285614
Enrollment number: 190170111075 Enrollment number: 190170111011
i
Acknowledgement
With great pleasure, we take this opportunity to express our deep sense of gratitude and
indebtedness to our external guide Shri Kirit Patel from Institute For Plasma Research
for his consummate knowledge, due criticism, invaluable guidance, and encouragement
which has enabled us to gain knowledge while carrying out this research project.
We are immensely thankful to our internal guides — Prof. Shreeji H. Shah and Prof.
Kalpesh B. Chaudhary — for their support and guidance throughout the course of the
project, thought-provoking and insightful questions during internal reviews, and their
invaluable comments on the report manuscript.
We are immensely thankful to respected HoD Dr. Arun B. Nandurbarkar, and respected
principal Dr. Nilay N. Bhuptani, for providing us the means to grab such an opportunity
to gain knowledge, as well as their everlasting willingness to extend their profound
knowledge, valuable support, and inspiration.
Finally, we would like to thank our friends and family for their support and patience
throughout, especially to our parents, without whose encouragement and support this
would not have been possible. Any attempt to define this indebtedness would be
incomplete.
– Siddh Raman Pant and Akhilesh Bahuguna
ii
Abstract
For many applications, like controlled plasma confinement in a tokamak (for controlled
tokamak fusion), a hard real-time distributed data acquisition and control system is
needed due to very large complexity of the systems (size of the machine, variety of
involved plant systems, etc.) under consideration.
The White Rabbit Project by CERN is a fully open-source effort intended in the same
direction. The White Rabbit Network features sub-nanosecond accuracy and picoseconds
precision of synchronisation for thousands of nodes over long Gigabit Ethernet fibre links.
This is a very promising feature set, and thus a closer look is necessitated.
In this project, we studied the White Rabbit project from a time synchronisation and event
distribution perspective. We also developed a time-based trigger / event distribution
utility to send triggers to different nodes, and measured, in various test configurations,
the time synchronisation accuracy along with general Ethernet data communication.
Hence, we get a good idea about the feasibility of deploying it in IPR.
In this report, we first introduce the requisite concepts and their application in White
Rabbit. Then the main White Rabbit gatewares used in the KIT SPEC card are introduced,
and then the requisite system apparatus and setup are detailed. We then demonstrate the
generation of a 1-PPS signal by a master-slave pair and compare it on an oscilloscope to
show the White Rabbit synchronisation practically. We then proceed to characterise how
we implemented trigger distribution, and then show the results.
iii
List of Figures
NUMBER CAPTION PAGE

Figure 1.1 New Research and Development Laboratory Building at IPR. 2
Figure 4.1 PTP process part 1: Calculating offset between clocks. 8
Figure 4.2 PTP process part 2: Calculating network latency. 8
Figure 4.3 Full PTP synchronisation flow. 9

Figure 5.1 Standard Ethernet v/s SyncE. 12
Figure 6.1 Phase tracking block diagram. 13
Figure 7.1 Addition of VLAN tag in an Ethernet frame, with structure. 15
Figure 8.1 A simple schematic of a distributed system, courtesy 18
GeeksforGeeks.
Figure 9.1 Steady State Superconducting Tokamak - 1. 20
Figure 10.1 White Rabbit PTP Core module and its interfaces. 23
Figure 10.2 WR-NIC HDL block diagram. 24
Figure 10.3 WR Streamers functional diagram. 25
Figure 10.4 EtherBone internals block diagram. 26
Figure 10.5 DIO core block diagram. 27
Figure 10.6 A rough architecture for implementing trigger distribution on 28
FPGA.
Figure 12.1 Two desktop PCs running Ubuntu 16.04 LTS. 33
Figure 12.2 A DSO-X 3034A oscilloscope showing a waveform. 34
Figure 12.3 White Rabbit Starting Kit, courtesy Seven Solutions. 34
Figure 12.4 A SPEC board, courtesy creotech. 35
Figure 12.5 An FMC-DIO card, courtesy creotech. 36
Figure 12.6 A LEMO 00 cable. 36
Figure 12.7 LEMO-BNC adapter. 37
iv
NUMBER CAPTION PAGE
Figure 12.8 AXGE-1254-0531 and AXGE-3454-0531 SFP transcievers. 37

Figure 12.9 A 5m fiber cable by HP. 38
Figure 12.10 White Rabbit Switch (front and back). 39
Figure 12.11 An RJ45 patch-cord. 40
Figure 13.1 A point-to-point WR link. 41
Figure 13.2 A WR network with WR switch. 42
Figure 13.3 A SPEC board (housing the FMC-DIO card) installed in a CPU. 43
Figure 14.1 SFP slot location on board. 47
Figure 14.2 WRPC shell GUI on slave before syncing Linux system's time. 48
Figure 14.3 WRPC shell GUI on slave after syncing Linux system's time. 49
Figure 15.1 A LEMO cable connected to channel 0 of the DIO. 50
Figure 15.2 Train of 1-PPS signals by master visible on a 1s/div scale. 51
Figure 15.3 1-PPS signals generated by master and slave on a 5ns/div scale. 52
Figure 15.4 The same 1-PPS signals as in the previous figure layered on top of 52
each other.
Figure 15.5 Persistence display of 1-PPS signals by master and slave on a 53

2ns/div scale.
Figure 16.1 Opening an SSH session on WRS. 55
Figure 16.2 Web management interface. 57

Figure 16.3 Active WRS ports. 58
Figure 18.1 Point-to-point trigger test's two trigger waveforms on a 10ms/div 64

scale.
Figure 18.2 The two trigger waveforms on a 2ns/div scale. Cursors are used to 65
get separation ( in the right box).
Figure 18.3 Oscilloscope display at the end of point-to-point multiple trigger 66
distribution test.
Figure 18.4 Switched single trigger test's waveforms on a 2ns/div scale. 67

Figure 18.5 Oscilloscope display at the end of switched multiple trigger 68
distribution test.
v
List of Tables
NUMBER TABLE PAGE
Table 8.1 List of control messages in MQTT (taken from MQTT reference). 31
Table 18.1 Pulse width issued in command v/s pulse width actually 63
generated, as seen on the oscilloscope.
vi
Table of Contents
TITLE PAGE
No Plagiarism Declaration i
Acknowledgement ii
Abstract iii
List of Figures iv
List of Tables vi
Table of Contents vii

1. Overview of IPR 1
1.1. Introduction 1
1.2. History 1
1.3. DACD Lab at IPR 2

2. Introduction to White Rabbit 3
3. Ethernet 4
3.1. Introduction 4
3.2. Working 4
3.2.1. Physical Layer 4
3.2.2. Auto-Negotiation 5
3.2.3. Encoding 5
3.3. Limitations 6
3.4. Real-Time Ethernet in White Rabbit 6
4. Precision Time Protocol 7
4.1. Introduction 7
4.2. Working 7
4.3. White Rabbit PTP 9
5. Synchronous Ethernet 11
vii
TITLE PAGE
5.1. Introduction 11
5.2. Working 11
5.3. SyncE in White Rabbit 12

6. Precise Phase Measurement 13
6.2. Working 13
7. Virtual Local Area Network 15
7.2. Working 15
7.3. Data Communication in White Rabbit 16
8. Distributed Systems 18
8.2. White Rabbit in Distributed Systems 19
9. Trigger Distribution 20
9.2. Need for Trigger Distribution 20

9.3. Trigger Distribution Using White Rabbit 21
10. White Rabbit Gatewares 22
10.1. White Rabbit PTP Core (WRPC) 22
10.2. WR-NIC 23
10.3. Network Fabric Interfaces 24
10.3.1. Plain Fabric 24
10.3.2. WR Streamers 25
10.3.3. EtherBone 25
10.4. DIO Core 26
10.5. An Approach for Trigger Distribution at FPGA Level 27
11. MQTT 29
viii
TITLE PAGE
11.2. Features 29
11.3. Working 30
11.3.1. Quality of Service 30
11.3.2. Control Messages 31
11.3.3. Last Will and Testament 32

12. Requisite Setup / Apparatus / Materials 33
12.1 Desktops with Linux Kernel v4.x 33
12.2. Digital Oscilloscope with Nanoseconds Level Zoom 34
12.3. White Rabbit Starting Kit 34

12.3.1. SPEC Board (x2) 35
12.3.2. FMC Digital I/O, 5 Channel, TTL A (x2) 36
12.3.3. LEMO 00 Connector (x3) 36
12.3.4. LEMO-BNC Adaptor (x3) 37

12.3.5. Bidirectional SFP Transcievers 37
12.3.6. Single-Mode Fibre, LC-LC (x2) 38
12.4. White Rabbit Switch 39
12.4.1. The WRS 39

12.4.2. RJ45 Patch-Cord 40
13. Initial Setup 41
13.1. Hardware Installation 42
13.2. Software Installation 43

13.2.1. Detection of the SPEC Card 43
13.2.2. Installation of Prerequisites 44

13.2.3. Main Installation 44
14. Configuring and Synchronising the WR Nodes 46
14.1. Virtual UART and Init Script 46

14.2. Master-Slave Connection 47
14.3. Synchronisation 47
ix
TITLE PAGE
15. Generation and Comparison of 1-PPS 50
15.2. Setup and Generation 50
15.3. Observation 51
16. Setting up White Rabbit Switch 54
16.1. Setting up a DHCP Server 54
16.2. Connecting WRS to DHCP Server 55

16.3. Upgrading WRS Firmware 56
16.4. Accessing the Web Management Interface 56
16.5. Setting up the PHY for White Rabbit Network 57

17. Trigger Distribution Implementation 59
17.2. Setup 60
17.3. Determining Resolution 61
17.4. Performance Testing 61

17.4.1. Point-To-Point 61
17.4.1.1. Single Trigger Distribution Test 61

17.4.1.2. Multiple Trigger Distribution Test 62
17.4.2. Network with WR Switch 62
17.4.2.1. Single Trigger Distribution Test 62

17.4.2.2. Multiple Trigger Distribution Test 62
18. Observation and Results 63
18.1. Resolution 63
18.2. Point-To-Point 64
18.2.1. Single Trigger Distribution Test 64

18.2.2. Multiple Trigger Distribution Test 65
18.3. Network with WR Switch 66
18.3.1. Single Trigger Distribution Test 66
x
TITLE PAGE
18.3.2. Multiple Trigger Distribution Test 67

19. Conclusion 69
References 70
xi
ㅤ Gujarat Technological University Team IDs: 318853, 285614
1. Overview of IPR
1.1. Introduction
Institute for Plasma Research (IPR) is an autonomous physics research institute located in Gandhinagar,
Gujarat, India. The research work at the institute revolves around plasma science, including plasma
physics, magnetic confinement of hot plasma, and plasma technologies for industrial applications. IPR
comes under the authority of Department of Atomic Energy (DAE). It is the leading plasma physics
organisation in India.
IPR is playing a crucial role in the Indian partnership in the international fusion energy initiative ITER
(International Thermonuclear Experimental Reactor). IPR is also a part of the IndiGO consortium for
research on gravitational waves. IPR owns two operational tokamaks - ADITYA and SST-1 (Steady State
Superconducting Tokamak). These tokamaks are used for the magnetic confinement of the high
temperature plasma.
1.2. History
The Institute for Plasma Research can trace its roots back to early 1970's when a coherent and interactive
programme of theoretical and experimental studies in plasma physics with an orientation towards
understanding space plasma phenomena was established at the Physical Research laboratory. In 1982, a
proposal to Government of India to initiate studies on magnetically confined high temperature plasma
was accepted which resulted in the establishment of the Plasma Physics Program (PPP) under the support
of Department of Science and Technology. In 1986, the Plasma Physics Program evolved into the
autonomous Institute for Plasma Research under the Department of Science and Technology. 1
With the commissioning of ADITYA in 1989, full-fledged tokamak experiments started. A dynamic
experimental programme focusing on transport due to edge turbulence has resulted in major discoveries
in this field. This period also saw development of new programmes in plasma processing and basic and
computational plasma research.
With the decision to move forward with the introduction of second generation superconducting steady
state tokamak SST-1 capable of 1000-seconds operation in the year 1995, the institute grew rapidly and
came under the authority of the Department of Atomic Energy. 1
IPR is now internationally recognised for its contributions to fundamental and applied research in plasma
physics and associated technologies.
ㅤ VGEC, Chandkheda Page 1 of 73

1.3. DACD Lab at IPR

At IPR, we carried out our project in the "Data Acquisition and Control Division" (DACD) located at the
"New R&D Laboratory".
Figure 1.1: New Research and Development Laboratory Building at IPR.
Various experimental activities at IPR need data acquisition and control in its systems. The DACD group
serves to address all these requirements at IPR.
DACD is actively exploring White Rabbit for implementing real-time trigger distribution, deterministic
data communication, and networking-related needs of IPR. Hence, to address these requirements, the
project we have worked on was conceived.

2. Introduction to White Rabbit
Many systems and applications, such as controlled plasma confinement in a tokamak (for controlled
tokamak fusion) or CERN's Large Hadron Collider (LHC), are distributed into several systems and nodes,
typically having a large complexity and size. Each node must be synchronised with each other in order to
have accurate and precise measurements and control, especially for events which last only for a short
period of time, otherwise the data acquired or control operation is rendered obsolete and may have serious
consequences. Thus, all the nodes in the system must not perform their operations randomly, but on a
fixed (synchronised) timing. To ensure synchronisation among all the distributed nodes, a time
synchronisation system is needed.
White Rabbit is a project initiated by The European Organisation for Nuclear Research (known as CERN),
and GSI Helmholtz Centre for Heavy Ion Research, along with several players from the industry, with the
aim of developing a distributed timing and data network in which a large number of nodes (on the order of
thousands) are synchronised with sub-nanosecond accuracy and picosecond precision relative to a master
timing station. 2
Previously at CERN, General Machine Timing (GMT) system was used for timing applications. But this
system had disadvantages such as limited bandwidth, unidirectional communication, and impossibility of
dynamically evaluating the link delay. To overcome these disadvantages, CERN went for Ethernet.
Currently, Ethernet is the most successful networking standard which has low costs for cabling and
infrastructure, high bandwidth, efficient switching technology and interoperability. 3 The data network
should have a deterministic behaviour with low transmission latency. Hence, the challenge of White
Rabbit is to accomplish high precision timing over Ethernet.
Synchronisation between the nodes is accomplished using the Precision Time Protocol (PTP) with
hardware-assisted timestamping. In order to reach high accuracy, White Rabbit minimises timing jitter by
using "data carrier frequency transfer" at the physical layer, referred to as Synchronous Ethernet (SyncE).
This ensures the syntonisation of master and slave node clocks, allowing for sub-nanosecond accuracy.
The sub-nanosecond accuracy is only achieved when both sides of the data link implement the White
Rabbit extension of PTP. Precise phase measurement using DDMTD allows for picosecond level of
synchronisation (i.e., the timestamp precision is in picoseconds). A tree structure topology is used to
distribute timing to a large number of nodes. 4
The White Rabbit project is a fully open source project. Both software and hardware HDL sources can be
accessed from the Open Hardware Repository (OHWR) website (ohwr.org).

3. Ethernet
3.1. Introduction
Ethernet is a communication protocol that connects several nodes (like computers, printer, scanner, etc.)
on a network over a wired connection. Ethernet is defined under the IEEE 802.3 standard 5 and is the most
widely used LAN technology.
It is important to understand Ethernet and its limitations, so that we can appreciate the use and
implementation of real-time Ethernet in White Rabbit.
3.2. Working
Ethernet operates on the physical and data link layer. Ethernet transmits the data in the form of frames
and packets. A data frame is composed of two components:
1. Header: It contains the physical / MAC addresses of the sender and receiver, along with error
correction data.
2. Payload: It contains the data / information to be transmitted.
3.2.1. Physical Layer

Ethernet supports multiple physical layers (PHYs). Any PHY is described in the following format (as
specified in section 1.2.3 of the standard): 5
1 <data rate> <modulation type> - <medium> <pcs encoding> <lanes (LAN) or reach
(WAN)>
One can find a definition list of PHYs on Section 1.4 of the standard. 5 10BASE-T, 1000BASE-TX,
1000BASE-F, are some examples.
Ethernet works on both full-duplex and half-duplex mode. Modern Ethernet networks operate in full-
duplex mode, with dedicated channels for transmission and reception. Several PHYs can be used, like
copper twisted pair cables and optical fibres, and speeds can go as high as 400 gigabits per second (Gb/s).

Legacy Ethernet operates on half-duplex mode, with nodes sharing the same medium. In this case,
multiple access contention is possible. It is arbitrated using Carrier Sense Multiple Access with Collision
Detection (CSMA/CD), in which all the nodes check availability of channel before sending the data by
sensing if there is any carrier / transmission ongoing on the channel. When no carrier is sensed, it means
no other station is probably transmitting, so the node starts the transmission. While transmitting, it
simultaneously checks the channel in the same way for collisions. If a collision happens, the node informs
the other nodes of the same by sending a jam signal, and backs-off for a random period of time. 5
3.2.2. Auto-Negotiation
Since multiple nodes and PHYs can be there in a network, it is possible that they have different operating
speed and communication scheme. Thus, for two nodes to communicate, they should do so on the
"highest" (i.e. more speed, full duplex, etc.) common mode possible between the two for maximum
advantage. This is done by the node by exchanging information ("advertising") about its capabilities, like
speed and duplex abilities, automatically. After the exchange, the nodes configure themselves
appropriately on the highest common mode possible. This process is (appropriately) known as auto-
negotiation, and is a mandatory feature for 1000BASE-T (i.e. for gigabit Ethernet over twisted pair) and
above.
3.2.3. Encoding
Clock can be encoded in the data using encodings like 4B/5B, 8B/10B, and 64B/66B. We will refer to them
as XB/YB.
In an XB/YB encoding, where X < Y, X bits are mapped to Y bits. That is, each X bit code is mapped to a
unique Y bit code. The mapping is done in such a way that the number of transitions are maximised, so
that clock recovery can be easily done by the receiver using a PLL. Special codes (commas) are added in
between for bit synchronisation.
Since Y > X, our effective bit rate increases. For example, for 1Gbps data rate using 8B/10B, we will have
Gbps data rate. This is a problem for communication over twisted pair, like in our specific
case the 1000BASE-T, since the copper medium is defined to support the maximum specified bitrate,
whereas in case of fibres, they generally support the higher bitrates.
Thus, we need to reduce the bitrate. This is done using techniques like MLT-3 (100BASE-T), 4D-PAM5
(1000BASE-T), PAM16 / DSQ-128 (for 10GBASE-T), where we use multiple voltage levels to encode the
data.
Thus, the process for 3 cases are as follows:
100BASE-T:
100 Mbps base data rate.

With 4B/5B, we will have Mbps data rate.
Using MLT-3 (having 3 voltage levels, thus 4 state transitions in a cycle), we will bring it
down to Mbps.
1000BASE-T:
1000 Mbps base data rate.
With 8B/10B, we will have Mbps (or 1.25 Gbps) data rate.
Using 4D-PAM5 (having 5 voltage levels, and thus 8 state transitions in a cycle), we will
bring it down to Mbps.
10GBASE-T:
10 Gbps (or 10000 Mbps) base data rate.
With 64B/66B, we will have Mbps data rate.
Using PAM16 / DSQ-128 (having 16 voltage levels, and thus 30 state transitions in a
cycle), we will bring it down to Mbps.
3.3. Limitations
Ethernet, and especially legacy Ethernet, is mainly used in a packet-switched network, which is inherently
asynchronous, i.e., the transmitter and receiver do not share an external clock signal. This means that the
transmission of data packets is random and collision between the data packets can occur which results in
packet loss. After collision, the back-off in CSMA/CD is for a random amount of time, thus the
transmission is not guaranteed to be done under a certain time (it is possible for the transmission to not
be done in case collisions keep happening after the wait). Further, there exists no priority among the
packets, and thus we are limited to FIFO best-effort communication. These things make the transmission
over Ethernet undeterministic in terms of latency and communication jitter.
Further, for encoded clock systems, the synchronisation is between the two communicating
endpoints/nodes, and not the entire network. The clock recovered at a node is not passed to the next node
(as the receiver and transmitter PHYs are different). That is, the synchronisation is between the hops, but
not passed from hop to hop. 6
3.4. Real-Time Ethernet in White Rabbit

The White Rabbit network runs over real-time Ethernet to achieve its objectives.
To have synchronisation over the network, PTP and SyncE is used. To have determinism, VLAN coupled
with appropriate network topology and cut-through implementation is used. We will talk about these in
the coming sections.

4. Precision Time Protocol
4.1. Introduction
We are interested in nodes being synchronised in time. Time synchronisation is a non-negotiable for us,
but we also want to use Ethernet. Any Ethernet network consists of a variety of network elements each
introducing some latency. Thus, the overall network will have a non-deterministic latency. This problem is
more serious for dynamic networks wherein network elements are added or removed in real-time.
Precision Time Protocol (PTP), defined currently by IEEE 1588-2019 (PTPv2.1), allows for time
synchronisation with a sub-ns precision throughout a network. 7 PTP can work over either packets
(network layer) or frames (link layer), and does not need any separate network for time synchronisation,
which reduces costs and network management overhead.
4.2. Working
PTP synchronises a slave's time to a master on the basis of timestamped packets (or frames)
communicated between them. The master and slave are determined dynamically on the basis of an
algorithm named Best Master Clock (BMC). 7 A slave for one node may function as a master for another
node. The entire network could synchronise time to a reference "grandmaster" clock, which has the
highest accuracy and precision.
In PTP lingo, a "clock" essentially refers to a device having a PTP compatible clock. In PTP, the following
three types of clocks are defined:
1. Ordinary Clock: The terminal devices.
2. Boundary Clock: The intermediary network elements.
3. Transparent Clock: The compensation for device latency of boundary clocks.
The masters and slaves will be ordinary clocks. Switches etc. in the network will function as boundary
clocks, having an incoming port functioning as slave, and outgoing ports functioning as masters. A
transparent clock will account for the inherent device delays that switches have in synchronising and
transferring the clock (timestamped packets/frames) from one port to the other, so that it will seem like
the switches do not contribute to delay (i.e. are "transparent") in the network.
The crux of the time synchronisation procedure is the following simple four-step procedure:

1. The master sends a SYNC message, and records the time at which the message is sent (say, ).
The slave receives it at some instant of time on slave's current clock (say, ).
2. The master sends a FOLLOW_UP message containing .
Master Slave
100 120
t1
101 SYNC 121
102 122
t2
FOLLO
103 W_UP 123
(100)
104 124
Figure 4.1: PTP process part 1: Calculating offset between clocks.
At this point, the slave has both and , and thus can calculate the offset between the master and slave
as , and then add it to the current clock. But the slave still has not accounted for network
latency.
3. The slave sends a DELAY_REQ , and records the time at which the message is sent (say, ). The
master receives it at some instant of time on master's current clock (say, ).
4. The master sends a DELAY_RESP message containing .
Master Slave
106 104
t3
107 _REQ 105

DELAY
108 106
t4
109 107
110 108
DELAY
111 _ RESP 109
(108)
112 110
Figure 4.2: PTP process part 2: Calculating network latency.

Now the slave can estimate the network latency as (assuming symmetric latency), and
then add it to the current clock.
Thus, the corrected (synced) clock can be expressed as:
Master Slave Information at slave
t1
SYNC
t2 t2
FOLLO
W_UP
(t )
1
t2, t1
t3 t2, t1, t3
_REQ
DELAY
t4
DELAY
_RESP
(t )
4
t2, t1, t3, t4
Figure 4.3: Full PTP synchronisation flow.
4.3. White Rabbit PTP
At the time of development of White Rabbit, IEEE 1588-2008 (PTPv2) 8 was in force, which did not
provide enough for the use cases needed by White Rabbit. White Rabbit is aimed at sub-ns
synchronisation, while the standard supported only sub-µs ranges. Thus, White Rabbit extended PTPv2
(known as WRPTP), and these efforts are now standardised in IEEE 1588-2019 (PTPv2.1). 9

Specifically, the following are the problems in PTPv2 which did not allow for the requisite accuracy: 10
1. Physical link (master ⇋ slave) is assumed to be symmetric, but in practical implementations

there is an inherent asymmetry, especially if using techniques like WDM for duplex
communication over single fibre (as in the KIT SPEC card).
2. Syntonisation depends on exchange rate of PTP messages. Having a high accuracy implies you
have to do PTP procedure very often so that the frequencies are matched and the master and
slave do not drift slightly. This will result in PTP traffic hogging the network.
3. Timestamp precision and resolution is limited at best by the hardware.
White Rabbit addresses them as follows 10 :
1. Delay asymmetry is calculated during link setup using the WR Link Delay Model.
2. Synchronous Ethernet is used for syntonisation at PHY (Layer 1) so that a common notion of
frequency exists.
3. Phase difference between the clocks is calculated using Digital Dual Mixer Time Difference
(DDMTD) with sub-picoseconds accuracy 11 , which allows for greater (picoseconds) resolution
and precision of timestamps.

5. Synchronous Ethernet
5.1. Introduction
Traditional Ethernet was primarily designed for transmission of asynchronous data which means there
was no requirement to pass any synchronisation signal between two nodes. 10Mbps (10BASE-T) Ethernet
is not even capable of sending any synchronisation signal because it stops sending pulses during the idle
periods. 10BASE-T transmitter sends a single pulse every 16ms to the receiver, to tell the the receiver that
it is still running. But this is not enough to achieve synchronisation of clocks between them. 6
Syntonisation is defined as synchronisation of frequency between two electronic circuits or devices.

Although syntonisation exists in traditional Ethernet between the transmitter and receiver on one link,
the recovered clock is not further propagated forward to other nodes of the network. 6
In order to achieve proper syntonisation in a network over Ethernet, a new mechanism named
Synchronous Ethernet (SyncE) was devised. SyncE is an ITU-T defined standard for ensuring that all the
nodes within a network are operating on the same clock frequency. SyncE sends the clock over layer 1
(PHY), along with the data but without affecting the data traffic, between the two nodes.
SyncE is standardised by ITU-T as three recommendations — G.8261 12 , G.8262 13 , and G.8264 14 .
5.2. Working
In SyncE, the network establishes a clocking hierarchy with the highest accuracy clock (the grandmaster)
at the top, followed by the slave nodes in the subsequent lower layers. Following are the clocks at different
levels of the hierarchy: 6
1. Primary Reference Clock (PRC): This clock has the accuracy of 10-11 which means for every 1011
pulses there will be one pulse more or less with reference to an ideal clock. PRC is generated from
a cesium clock or from another source of clock like GPS (Global Positioning System).
2. Synchronisation Supply Unit (SSU) or Building Integrated Timing Supply (BITS): This is the
second level in the hierarchy. SSU/BITS has a feature named Holdover, that allows it to produce a
higher accuracy clock than its intrinsic free running accuracy for short period of time after losing
connection with the PRC. SSU/BITS is implemented using Digital PLL run by Rubidium clock.
3. SDH Equipment Clock (SDH) or SONET Minimum Clock (SMC): This is the third level in the
hierarchy. SDH/SMC also implements holdover, but it has lesser accuracy than SSU/BITS. It is
implemented with the digital PLL driven by the Temperature Controlled Oscillator (TCXO).

Figure 5.1: Standard Ethernet v/s SyncE. 15
The way all the nodes in the network gets synchronised to the same clock frequency is as follows:
1. The clock generated by the master is encoded in the bitstream and sent to all the slave node or
switches through the downlink port.
2. Slave nodes receive this bitstream through the uplink port and extract the clock frequency from it
with the help of Clock and Data Recovery (CDR) block.
3. This clock frequency is then locked in the receiving node oscillator with the help of Digital Phase
Locked Loop (DPLL).
The supported Ethernet interfaces (PHY, IEEE clause, coding) for SyncE can be seen in Appendix III, Table
III.1 of ITU-T Rec. G.8262. 13
5.3. SyncE in White Rabbit

White Rabbit is implemented using Gigabit Ethernet (GbE) on fibre. The advantage of GbE is that the link
is never idle. Even if the transmitter has nothing to transmit, the Medium Access Control (MAC) block
generates special 8B/10B patterns, which causes continuous transitioning pulses, to the receiver.
The continuous transmission of the special blocks with the transitioning pulses allows for clock recovery
at the receiver using SyncE, and it ensures that the DPLL on the CDR circuit does not gets unlocked. 3

6. Precise Phase Measurement
6.1. Introduction
To achieve complete synchronisation among the nodes of a hard real-time distributed system, it is
important to ensure time, frequency as well as phase synchronisation. Synchronous Ethernet is
responsible for clock syntonisation across all the nodes in White Rabbit Network, causing all nodes to
operate on the same frequency. Hence, we can evaluate delays by means of calculating the phase
difference among the clock signals of the nodes, and then compensate for those delays by correcting the
phase offset which results in phase synchronisation. A Digital Dual Mixer Time Difference (DDMTD) can
be used to calculate phase difference with sub-picoseconds accuracy. 11
The reinterpretation of the problem of link delay to that of phase measurement enables the White Rabbit
Network to achieve picosecond level of precision for its timestamps. 10
6.2. Working
An abstract description of the operations performed between the master node and slave node for phase
synchronisation is as follows:
Figure 6.1: Phase tracking block diagram. 3
1. In Synchronous Ethernet, we have seen that the master node sends the encoded bitstream to the
slave node. The slave node extracts the clock from the bitstream and cleans it by removing any
jitter present with the help of digital PLL.
2. This clock signal is encoded by the slave node and returned back to the master node. Thus the
master node will have original reference clock as well as the delayed copy of the reference clock.

3. These two clock signals are fed to the Digital Dual Mixer Time Difference (DDMTD) block inside
the master node, which measures the phase difference between them. The phase difference
between these two clock signals is the result of the link delay. The calculation of this phase
difference can be included in the PTP equation to have better precision.
4. This measured phase difference is sent using the PTP message to the slave node.
5. Inside the slave node, the DDMTD based Phase Shifting PLL block will correct the phase offset of
the slave clock to compensate for the link delay, resulting in phase synchronisation between the
master and slave node.
In this way, White Rabbit nodes calculates the phase difference with sub-picosecond accuracy, and we get
picosecond level of precision for our timestamps.

7. Virtual Local Area Network
7.1. Introduction
Normally, a switch can send messages in either a unicast (the message being sent to a specific port) or a
broadcast (the message being sent to all ports) fashion. But what if we want to send a message to a group
of devices, i.e., on a set of ports, but not all ports? We would want to do this because a controller may send
commands to a group of sensors or actuators. A similar problem is communicating to a set of nodes spread
across a network, handled by different and/or chained intermediary switches. These problems are our
topic of discussion because distributed data acquisition and control systems are inherently facing these
problems due to the nature of system design, and for hard real-time systems we need a deterministic way
of ensuring delivery of control messages.
A Virtual Local Area Network (VLAN), as the name suggests, allows us to make "virtual" partitions or
groups such that the grouping functions like a separate LAN. That is, the network is segmented into
different broadcast subdomains. This segmentation is achieved by use of identifiers or tags for each
subdomain, which allow network equipments (like a switch) to determine whether the message is to be
sent to a device (port) or not. On Ethernet, this tagging is done on the data link layer, i.e. added to the
Ethernet frame, and is defined by IEEE 802.1Q standard. 16
7.2. Working
Destination MAC
Destination MAC address (6 bytes)
address (6 bytes) Source MAC address
Source MAC address (6 bytes)
TPID (2 bytes)
(6 bytes) 802.1Q (VLAN) tag PCP (3 bits)
Length / Type (4 bytes)
TCI (2 bytes) DEI (1 bit)
(2 bytes) Length / Type
(2 bytes) VID (12 bits)
Payload (padded)
(Integral bytes) Payload (padded)
FCS (CRC) (Integral bytes)
(4 bytes) FCS (CRC)
(4 bytes)
Figure 7.1: Addition of VLAN tag in an Ethernet frame, with structure.
Tagging is done by adding a 4 byte (32 bit) VLAN tag in the original Ethernet frame, between the source
MAC address and length fields. The VLAN tag consists of the following fields: 16

1. Tag Protocol Identifier (TPID), which stays in the overall frame at the place where originally
the EtherType field was present. It contains an EtherType value used to identify the frame as a
tagged frame, so that the device can process it appropriately. The value can be among {0x8100,
0x88a8, 0x88e7} (the latter two are meant for use by service providers).
2. Tag Control Information (TCI), containing the following control fields (in case TPID is 0x8100):
a. 3 bit Priority Code Point (PCP), indicating quality of service for frame prioritisation.
b. 1 bit Drop Eligible Indicator (DEI), indicating whether the frame can be dropped in
case of congestion / high traffic.
c. 12 bit VLAN Identifier (VID), ranging from 0 to 4095. 0 (0x000) and 4095 (0xFFF) are
reserved. 0 indicates that this frame is to be simply prioritised and it does not belong to
any specific VLAN.
On reception of a tagged frame, the switch can determine where (i.e. on which port) and how (i.e.
prioritising, untagging) to send the frame. A switch which supports VLAN can encounter two scenarios,
and thus have the following two types of ports:
1. Access port: This port has a node directly connected to it. Thus, this port can be a part of just a
single VLAN.
2. Trunk port: This port has another switch connected to it. Since the other switch can have
multiple access ports, this port can be a part of multiple VLANs.
When sending frames to an access port (or potentially multiple access ports, in case of a broadcast in a
VLAN), the switch removes the VLAN tagging and sends untagged Ethernet frames to the nodes, since the
node may or may not have support for VLANs. This is an added advantage — the use of VLANs is
transparent to the end nodes.
A network-wide VLAN can thus be created by a combination of access and trunk ports. Nodes belonging to
same VLAN may be located in different areas, and with their associated switches connected to trunk ports
of other switches (or each other). Thus, a chained network can be created, and nodes in a VLAN can be
dynamically added. The standard defines several bridge management and routing protocols to enforce the
VLAN network.
7.3. Data Communication in White Rabbit

A certain degree of determinism can be achieved by use of priority frames. The smaller is the frame and
the number of nodes transmitting the highest priority traffic, the more deterministic is the latency, as the
highest priority traffic will be less common (or occupy less proportion of network traffic), and hence will
be more easily prioritised. The best case corresponds to just a single node acting as source of highest
priority frames, with a fixed (or minimum) size of frames. 17

The error checking also eats up our time. In a store-and-forward implementation, the incoming frame will
first be stored, get it checked using CRC, and will be forwarded only if the CRC check passes. In a cut-
through implementation, the CRC is not checked and is forwarded as soon as the destination is known (i.e.
upon reception of header). Transmission of one frame must complete before transmitting another frame,
even if it is high priority. Thus, a variable size of frames can be problematic if we are considering
determinism.
White Rabbit provides deterministic data forwarding, which is possible by building on top of a tree-based
architecture which uses VID to identify ports belonging to optimal paths (known as active ports), and
discard frames on other (backup) ports. The path and ports are pre-configured, and in case the active port
goes down, the backup port will be used (the switchover is fast). Thus there is redundancy and latency
minimisation.
White Rabbit classifies data into two types:
1. Best-effort data: The normal way — Ethernet traffic delivered on a best-effort basis.
2. Critical data: The critical traffic meant be delivered deterministically.
The data traffic is identified on the basis of priority in the VLAN tag. White Rabbit prioritises critical data
over best-effort data by means of deterministic cut-through. It functions almost like the normal cut-
through discussed above. The change is that if a best-effort frame is being transmitted when a critical
frame has to be, the transmission of the best-effort frame is stopped immediately (the CRC is made invalid
so the destination discards it) and the critical frame is transmitted.
This makes the critical data latency independent of the standard Ethernet traffic, and thus a degree of
determinism can be achieved by way of careful implementation of hardware (switches) and network
topology (number of critical data sources). 17

8. Distributed Systems
8.1. Introduction
Distributed systems is defined as the group of autonomous computing elements that works altogether to
achieve a common goal and appears to be a single coherent system to its user. Each computing element in
the distributed system is able to behave independently of each other. These computing elements, referred
to as nodes, can be either a hardware device or a software process. In order for these collection of nodes to
appear as a single system, there must be establishment of collaboration among these nodes.
Distributed systems can consist of range of nodes, from a highly computational machine to small sensor
device. All the nodes can act independently from each other, but they do not ignore each other else there
is no use use of putting them into a system. In practice, all the nodes communicate with each other by
exchanging messages. A node reacts to an incoming message, which are then processed, leading to further
communication through message passing. As the nodes are independently working, each one them will
have its own notion of time meaning there won't be any global clock. This could raise issues of
synchronisation and coordination in a distributed system.
Figure 8.1: A simple schematic of a distributed system, courtesy GeeksforGeeks. 18
As distributed system is a group of nodes, we need to manage the membership and organisation of the
group. It means we need to ensure whether a node belongs to the system and to which nodes it can
communicate with. Based on this, we can define open group and closed group, as follows:

Open group: Any node is allowed to join a distributed system and allowed to communicate with
the other nodes of the system.
Closed group: Only members of the group are allowed to communicate with each other and a
separate mechanism is required for a node to join or leave the system.
Since distributed systems masquerades a number of spatially separated (and possibly complex) systems as
one single system, resources can be added, shared, and removed dynamically. Also, due to constituent
systems themselves being separate, if the isolation is properly configured, then if one system fails it is
unable to take down the whole system, that is, we have a degree of fault tolerance.
The distribution over network introduces a significant amount of latency when compared to a single
system at one place / connected physically nearby. One must also make sure to have the systems interact
and coordinate properly, i.e., synchronisation should be taken care of.
8.2. White Rabbit in Distributed Systems

We know that White Rabbit is a distributed timing and data network in which a large number of nodes are
synchronised with sub-nanosecond accuracy and picosecond level precision relative to a master timing
station / grandmaster. Thus, the local oscillators of all the nodes in the network run with respect to the
same reference clock and all the operations performed among the nodes will be coordinated.
Also, White Rabbit networks feature determinism, i.e., there is a fixed upper bound for network latencies.
This is done by making use of VLAN and sending priority frames between the nodes, as explained earlier.

9. Trigger Distribution
9.1. Introduction
A trigger is basically a pulse signal that is used to initiate an action in an electronic device. Triggers are
generally used to initiate processes in a synchronised way among various nodes in a control network. We
have seen that in distributed systems, one of the major constraints is to make all the nodes synchronised
with each other to a common clock.
Trigger distribution is the process of distributing triggers on a network. This can involve setting up
triggers on multiple devices and configuring the devices to work together seamlessly. It can be done in
several ways, but the one that we have gone for is by using real-time Ethernet network based on White
Rabbit.
9.2. Need for Trigger Distribution

Trigger distribution is an important application in large (in both complexity and geography) distributed
systems. For example, at IPR, a device named Steady State Superconducting Tokamak - 1 (SST-1) is used to
create hot plasma and confine it with the help of magnetic field, to carry out various physics related
studies in the domain of plasma physics and nuclear fusion.
Figure 9.1: Steady State Superconducting Tokamak - 1. 19

Tokamak, with a number of supporting subsystems, is a sophisticated distributed system that requires all
the necessary events throughout the process to occur at accurate and precise timings. Even a small delay
in any event can cause the failure of the whole process.
An abstract description of the working of SST-1 (from the point of view of triggers) is as follows:
1. A trigger is given to a node that puffs gas inside the tokamak at a fixed pressure.
2. Then another trigger is given to a different node that injects electromagnetic wave in the
tokamak to ionise the gas.
3. Again a trigger is given to a different node that passes the current through the central solenoid to
create a magnetic field inside the tokamak.
4. The plasma is confined within the magnetic field lines and circulates in the toroidal chamber
inside the tokamak.
Thus, we can observe that the distribution of triggers among the various nodes of the tokamak becomes
really important to carry out the whole experiment successfully, and thus synchronisation must be
maintained among them. The current accuracy of trigger distribution system in place is on the order of
milliseconds.
9.3. Trigger Distribution Using White Rabbit

As we have already discussed, White Rabbit allows for sub-ns synchronisation and deterministic data
communication. In a WR network, all the WR nodes will be synchronised to the master clock with the help
of PTP and SyncE. That is, the time and frequency of all the nodes will be the same.
We can thus build upon the clock synchronisation of nodes in a WR network to distribute triggers. Triggers
distribution will involve sending the time at which the trigger pulse needs to be generated. Since all the
nodes in a WR network will have the same clock and frequency, the triggers will be generated at the exact
time given, and hence the activities of multiple nodes in a data acquisition and control system can be
adjusted and synchronised.

10. White Rabbit Gatewares
10.1. White Rabbit PTP Core (WRPC)

White Rabbit PTP Core is the IP core present inside the FPGA of SPEC card that has single gigabit ethernet
interface and capable of providing precise timing. Currently, WRPC supports 100BASE-X optical link.
WRPC is used to send and receive messages in the form of ethernet frames from the user-defined HDL
modules to the physical medium. This communication occurs through the SFP of the SPEC card using the
optical fibre link.
The White Rabbit PTP Core can be operated in the following modes:
1. Grandmaster: A WR node configured in grandmaster mode has the most accurate clock. For this
mode, external 10 MHz and 1-PPS signal (from GPS / cesium clock) is required to be given as
input to the grandmaster. Grandmaster propagates the precise timing clock to other WR nodes.
2. Master: A WR node configured in master mode has free running oscillator and propagates the
precise to the other WR nodes.
3. Slave: A WR node configured in slave mode takes the precise timing from the master and locks
its oscillator in synchronous with master's oscillator.
White Rabbit PTP Core has mainly four interfaces that are briefed below 20 :
1. Ethernet MAC interface: WRPC has MAC interface that is used for sending and receiving the
non-PTP Ethernet frames to/from the NIC (Network Interface Card) block, using two pipelined
Wishbone interfaces (master for forwarding and slave for receiving). It also provides the user
application with timestamps of every incoming and outgoing packets.
2. Wishbone Slave: It is used to provide access to the internal control registers and loading the
embedded CPU firmware.
3. Timing Port: WRPC can pass 1-PPS, TAI (International Atomic Time) timecode and 125 MHz
reference frequency through the timing port. Depending on the mode of operation, it can input
the external reference signal or it can output the recovered signal.
4. Control/Status Pins: These are the GPIO (General Purpose Input/Output) lines that provides a
basic control interface and are used to report the operation state of WRPC .
Note: At some places in the documentation and code files, TAI timecode is referred as UTC (Coordinated
Universal Time) timecode.

Figure 10.1: White Rabbit PTP Core module and its interfaces. 20
WRPC (before v5) consists of a soft-core processor named LM32 (LatticeMico 32), that has 32-bit RISC
architecture. Since v5, the soft-core is a RISC-V processor (with an option for LM32). It manages all the
internal modules of the WPRC and runs the White Rabbit PTP software synchronisation daemon. It is also
used to setup virtual UART, for serial communication.
10.2. WR-NIC
White Rabbit Network Interface Card provides the interface for the network communication between the
host (user application) and White Rabbit PTP Core block. WR-NIC makes the combination of the SPEC and
FMC-DIO card acts as a network interface card under Linux.
When incoming ethernet frames are received by the WRPC block from the White Rabbit network through
fiber cable, WR-NIC interrupts the host and provides it with a descriptor which is used by the host to fetch
the incoming ethernet frames. For outgoing Ethernet frames, WR-NIC receives the descriptor from the
host and fetches the Ethernet frames from the PCIe through the Gennum (GN4124) Core block (it acts as a
bridge between the PCIe interface card and the internal wishbone bus). After that, WR-NIC sends the
Ethernet frames to the WRPC block through the pipelined wishbone interface. 21
There are various other blocks that are essential. The Wishbone Interconnect block is used to ensure
proper interconnection among the wishbone masters and slaves using the crossbar topology. The TxTSU
block collects the timestamps of the associated Ethernet frame identifiers and put them in a shared FIFO.
An interrupt request is generated whenever this FIFO is not empty so that the drivers could read the
transmitted timestamps and frame identifiers.
Network interfacing is plain — WR-NIC driver generates Ethernet frames which are then passed to the
WRC. EtherBone is supported, but is disabled by default. More on network interfacing in the next section.

Figure 10.2: WR-NIC HDL block diagram. 21
10.3. Network Fabric Interfaces

The g_fabric_iface generic parameter in entity xwrc_board_common (in the file wr-
cores/board/common/xwrc_board_common.vhd ) determines the block to attach to the WRC fabric
interface. This interface is responsible for network communication through WRC via WR network. By
taking advantage of WR network and accounting for short network latencies by delaying the transmission,
these interfaces can guarantee fixed transmission latency.
It is important to understand the interface for implementing trigger distribution in HDL in future. There
are three possible configurations (achieved by setting respective values in g_fabric_iface ), as follows:
10.3.1. Plain Fabric

This is the default configuration, with g_fabric_iface = PLAIN . The vanilla WRC fabric interface is
exposed, wherein both input (for transmission) and output (for reception) are in the form of raw Ethernet
frames.

10.3.2. WR Streamers
WR Streamers provides word level abstraction over the Ethernet interface. One simply has to pass the
words (word size is configurable) appropriately, WR Streamers will take care of encapsulating the words in
Ethernet frames and passing it to WRC for the transmission. It also handles reception — it will unpack the
received Ethernet frames and output the words received.
WR Streamers works in a FIFO fashion (hence the name). This interface is ideal for VHDL modules at a
node which can just send data words to one block and have it transmitted over the WR network. The same
data will be received at the other node in same fashion. 22
Figure 10.3: WR Streamers functional diagram. 22
WR Streamers can be enabled by setting g_fabric_iface = STREAMERS . WR Streamers also provides

advanced diagnostics and debugging capabilities that can be accessed via SNMP, Wishbone registers (via
PCI or VME), and WRPC shell. 22
10.3.3. EtherBone
EtherBone provides remote bus interface over Ethernet. That is, we can connect two Wishbone (WB) buses
on different nodes via Ethernet (hence the name). Thus, we extend the bus over the network. EtherBone
can be enabled by setting g_fabric_iface = ETHERBONE .
EtherBone provides a pipelined bus-cycle interface (we synchronise to bus cycles). Whenever we have to
do a read/write operation on the remote WB bus, we write on the local EtherBone bridge via the local WB
bus / interconnect. The EtherBone core will take care of transmitting the operation over the network to
the remote core, wherein upon reception of the request, the EtherBone core at the remote node will

execute the operation over the bridge, with the WB interconnect routing it to the correct address. 23
Figure 10.4: EtherBone internals block diagram. 23
10.4. DIO Core

The Digital Input Output (DIO) core is an HDL block that access to the FMC-DIO-5ch mezzanine card. The
DIO core is responsible for the configurations of all the five channels that are present in the FMC (FPGA
Mezzanine Card) DIO card.
For input signals, it provides accurate TAI timestamps, taken from the WRPC block, as well as the host
interrupt through the IRQ Gen (Interrupt Request Generator) block (it takes one-tick long pulse from all
the other blocks and generates interrupt requests to the GN4124 core).
For output configuration, DIO allows the user to generate pulses at any future TAI time instant, or to
generate pulse immediately. It takes parameters of the pulse from the user such as pulse width, period and
count. Also it provides the accurate TAI timestamp of the generated pulse.
The FMC-DIO kernel driver consists of a set of functions that are accessible through an IOCTL (Input
Output Control) call to read and write the registers of each channel of the FMC-DIO board to perform
various operations and configurations.

Figure 10.5: DIO core block diagram. 21
The submodules present inside the DIO core are: 24
1. GPIO: General Purpose Input/Output core has many pins that can be used for digital
input/output signals. These pins can be configured as input/output and with/without tri-state
buffer, to perform read/write operations.
2. One wire: This core is used for the acquisition of the temperature information.
3. I2C: This core allows to set the threshold value of the comparator inside the DIO mezzanine and
provides access to read/write data to the EEPROM memory.
4. Pulse Generator: This core produces a programmable pulse in its output when the time instant
passed to it matches with the WRPC time (DIO takes this time from WRPC).
5. Pulse Stamper: This core is used to associate a time tag with the input/output pulse.
10.5. An Approach for Trigger Distribution at FPGA Level

We came up with a rough architecture for distributing triggers using custom HDL module and interfacing
(see figure 10.6) after our study, and side-by-side were exploring software approach. We went for the
software approach as a first pass, as it allowed for fast prototyping.
Still, HDL level implementation is attractive as a final solution since it runs fully on the FPGA. This is thus
a definitive future scope.

PCIe Bridge
Linux GN4124
System Core
(Driver needed)
Wishbone Bus
Diagnostics Time to
Trigger
Custom Trigger (Rx)

FMC-DIO
HDL Digital I/O
Core
Module
Rx Tx
EtherBone
or
WR Streamers
Reference
Clock Ethernet
(125 MHz) Frames
WR PTP WR
SFP
Core Network
Time Signals
Figure 10.6: A rough architecture for implementing trigger distribution on FPGA.

11. MQTT
11.1. Introduction
MQTT is an open application layer message transport protocol working on publish-subscribe model. The
clients can subscribe to and publish on "topics" (an abstraction over filtering of broker, using strings as
identifiers). The broker handles the distribution of messages to the interested parties upon their
publishing. The latest version is MQTTv5.
The MQTT reference summarises it very well as follows: 25
MQTT is a Client Server publish/subscribe messaging transport protocol. It is light weight, open,
simple, and designed to be easy to implement. These characteristics make it ideal for use in many
situations, including constrained environments such as for communication in Machine to Machine
(M2M) and Internet of Things (IoT) contexts where a small code footprint is required and/or network
bandwidth is at a premium.
Clients can subscribe to several topics, and they will receive messages only for the topics they subscribe to.
MQTT allows for wildcard characters and creation of topic hierarchies. Thus, a network can be
appropriately segmented. In a master-slave type of network, the master nodes (publishers) can easily send
messages to several slave nodes (subscribers) at once.
MQTT has a wide support and libraries are available for all major languages like C, Python, etc. Thus, the
libraries abstract away the low-level workings of MQTT in applications. But it is a good idea to have an
understanding about MQTT so that one can know what they are dealing with, implement and use the
libraries appropriately, and debug easily.
11.2. Features
The following are the features of MQTT, taken directly from the reference: 25
The protocol runs over TCP/IP, or over other network protocols that provide ordered, lossless, bi-
directional connections. Its features include:
Use of the publish/subscribe message pattern which provides one-to-many message distribution
and decoupling of applications.
A messaging transport that is agnostic to the content of the payload.
Three qualities of service for message delivery:

"At most once", where messages are delivered according to the best efforts of the operating
environment. Message loss can occur. This level could be used, for example, with ambient
sensor data where it does not matter if an individual reading is lost as the next one will be
published soon after.
"At least once", where messages are assured to arrive but duplicates can occur.
"Exactly once", where messages are assured to arrive exactly once. This level could be used,
for example, with billing systems where duplicate or lost messages could lead to incorrect
charges being applied.
A small transport overhead and protocol exchanges minimized to reduce network traffic.
A mechanism to notify interested parties when an abnormal disconnection occurs.
11.3. Working
As mentioned earlier, client and broker are the two network entities central to MQTT. A MQTT client is a
device which supports MQTT protocol and can connect to an MQTT broker. An MQTT broker is a server
which handles connection of clients, and filtering and distribution of messages. The client connects to
broker typically using TCP/IP.
MQTT clients do not communicate by use of addresses of other clients. MQTT uses "topics", a string (like
the "subject line" of a letter), for distribution of messages. The publishers publish messages on a "topic",
i.e., when the publisher client sends the payload to the broker server, it specifies the topic in the payload.
The broker then broadcasts the received message to only those subscribers who have subscribed to that
topic.
Since the broker does the filtering based on topic and the eventual message distribution, MQTT's publish-
subscribe model allows for decoupling of clients. The subscriber clients can subscribe to several topics
independently without the knowledge of the publishers. The publisher can publish without the knowledge
of subscribers. The broker is the one who has the knowledge of publishers, subscribers, and topics, and
appropriately handles the distribution.
11.3.1. Quality of Service (QoS)

As mentioned earlier, there are three QoS levels:
QoS 0: At most once — best effort delivery with no acknowledgement.
QoS 1: At least once — A simple two-part handshake for the acknowledgement of reception.
QoS 2: Exactly once — A four-part handshake mechanism for acknowledgement and ensuring
that the message is received only once.

Thus, the overhead increases as we increase the QoS level. Note that QoS is between the client and server,
i.e., between the broker and publisher or between the broker and subscriber. The QoS is not between
publisher to subscriber (though seemingly convenient, it would impinge on the decoupling). If the
subscriber specifies a lower QoS (compared to the published message) when connecting, the message sent
by the broker to subscriber is sent with the lower QoS.
11.3.2. Control Messages

An MQTT connection is established using the CONNECT message. A CONNACK messge is sent by the broker
acknowledging the connection.
A publisher can publish using a PUBLISH message. Depending on QoS level, there may or may not be
acknowledgement and further control messages. In MQTTv5, one can also add custom metadata / define
key-value pairs of properties.
A subscriber can request to subscribe to a topic using a SUBSCRIBE message. The broker acknowledges the
subscription with a SUBACK message.
Similarly, an unsubscribe request can be sent using a UNSUBSCRIBE message. The broker acknowledges it
with an UNSUBACK message.
The full list of control messages is given in table 11.1.
Table 11.1: List of control messages in MQTT (taken from MQTT reference). 25
NAME VALUE FLOW DIRECTION DESCRIPTION
Reserved 0 Forbidden Reserved
CONNECT 1 Client to Server Connection request
CONNACK 2 Server to Client Connect acknowledgment
PUBLISH 3 Client ⇌ Server Publish message
PUBACK 4 Client ⇌ Server Publish acknowledgment (QoS 1)
PUBREC 5 Client ⇌ Server Publish received (QoS 2 delivery part 1)
PUBREL 6 Client ⇌ Server Publish release (QoS 2 delivery part 2)
PUBCOMP 7 Client ⇌ Server Publish complete (QoS 2 delivery part 3)
SUBSCRIBE 8 Client to Server Subscribe request
SUBACK 9 Server to Client Subscribe acknowledgment
UNSUBSCRIBE 10 Client to Server Unsubscribe request
UNSUBACK 11 Server to Client Unsubscribe acknowledgment

NAME VALUE FLOW DIRECTION DESCRIPTION
PINGREQ 12 Client to Server PING request
PINGRESP 13 Server to Client PING response
DISCONNECT 14 Client ⇌ Server Disconnect notification
AUTH 15 Client ⇌ Server Authentication exchange
11.3.3. Last Will and Testament (LWT)

We may have clients suddenly getting disconnected — i.e., we may have ungraceful disconnections. This is
undesirable, hence there should be a way to know about it. MQTT allows this by use of the LWT feature.
The "Will" flag in the CONNECT header must be set to 1 to enable this feature.
A client can specify its will when it connects to the broker. The "will" consists of (refer section 3.1.3 of
MQTT reference 25 ):
1. Will Properties
a. Property Length
b. Will Delay Interval
c. Payload Format Indicator
d. Message Expiry Interval
e. Content Type
f. Response Topic
g. Correlation Data
h. User Property
2. Will Topic
3. Will Payload

12. Requisite Setup / Apparatus / Materials
The following are the things required for performing the experiments conducted by us and mentioned in
this report.
12.1. Desktops with Linux Kernel v4.x
Figure 12.1: Two desktop PCs running Ubuntu 16.04 LTS.
At least two desktops with good internet connection are needed. They must run Linux kernel v4.x, as the
drivers we are going to install are made for that version. Recommended distro is Ubuntu 16.04 LTS. The
drivers have been tested on it, and the user manuals recommend using that version.
We had tried using newer distros initially, but eventually used Ubuntu 16.04 LTS. 16.04 runs on kernel v4.x
by default, which is the primary reason. One can also try installing older kernel on newer distros, which
will work, but its best to go with dual-booting, as having a separate OS will also give a degree of freedom
for any messups.
Basic programs and build tools must be installed (like apt , gcc , etc.). If you use the recommended distro,
they come by default. Also, please make sure any proxy settings are appropriately set up for the command
line tools to connect properly. For shared PCs, you can find some helpers for bash in the following gist:
https://gist.github.com/siddhpant/0c5dd3d5bf4762b0d4c4bb2ac75026f7

12.2. Digital Oscilloscope with Nanoseconds Level Zoom
Figure 12.2: A DSO-X 3034A oscilloscope showing a waveform.
The oscilloscope should at least allow zooming in to 2 ns/div (horizontal scale). The oscilloscope must
have a trigger feature, which is really a standard nowadays. If the scope has cursors, it will be very helpful.
We used InfiniiVision DSO-X 3034A 350 MHz oscilloscope, which provides 4 analog channels, up to 4 Mpts
memory, and 1,000,000 waveforms/sec update rate. 26
12.3. White Rabbit Starting Kit
Figure 12.3: White Rabbit Starting Kit, courtesy Seven Solutions. 27

The White Rabbit Starting Kit can be purchased from Seven Solutions (KIT SPEC-S 28 ). It consists of the
following things:
12.3.1. SPEC Board (x2)
Figure 12.4: A SPEC board, courtesy creotech. 29
The Simple PCI Express FMC Carrier (SPEC) is a board that holds an FMC (FPGA Mezzanine Card) and an
SFP connector. It has a 4 lane PCIe interface that can be connected to x4, x8, or x16 port on the PCIe slot
of the motherboard of any PC.
SPEC board has one "Xilinx Spartan 6 XC6SLX45T" FPGA, which is the central part of the board housing
WR cores. A Gennum GN4124 core provides the bridge between the PCIe interface and the local buses in
the FPGA. SPEC also has Voltage Controlled Oscillators (VCXOs) that are required to produce a reference
frequency for DDMTD phase detection. On-board memory includes one 2GB DDR3, and one SPI 32MB
flash PROM for multiboot FPGA powerup configuration. For standalone features, an external 12 volt power
supply connector, 4 LEDs, 2 buttons, and one mini-USB connector together with a USB-UART bridge
(convenient for outputting debug messages from the design) is provided.
More information can be found on https://ohwr.org/project/spec/wikis/home.

12.3.2. FMC Digital I/O, 5 Channel, TTL A (x2)
Figure 12.5: An FMC-DIO card, courtesy creotech. 30
This FMC card gets fitted into the SPEC board, and provides 5 channel programmable digital I/O by the
use of LEMO 00 connectors. More information can be found on https://ohwr.org/project/fmc-dio-5chttla/w
ikis/home.
12.3.3. LEMO 00 Connector (x3)
Figure 12.6: A LEMO 00 cable.

A LEMO 00 cable is needed for Digital I/O with the FMC-DIO card.
12.3.4. LEMO-BNC Adapter (x3)
Figure 12.7: LEMO-BNC adapter.
We use LEMO connectors for Digital I/O, but BNC connectors have to be used to connect to oscilloscope.
Thus, a LEMO-BNC adapter is needed.
12.3.5. Bidirectional SFP Transcievers
Figure 12.8: AXGE-1254-0531 and AXGE-3454-0531 SFP transcievers.

Two bidirectional SFP transcievers are needed for optical communication over a single-mode fibre. We use
AXGE-3454-0531 31 (with violet clip) and AXGE-1254-0531 32 (with blue clip) SFP transceivers. The fibre
will be connected where the black cap is present in the above photo.
Note that both of them differ in wavelength — this is crucial, since we will be using a single mode fibre for
full-duplex communication, so wavelengths need to be different for uplink and downlink (WDM concept).
The violet-coloured SFP is an OLT transciever, and its laser has a wavelength of 1490 nm. The blue-
coloured SFP is an ONU transciever, and its laser has a wavelength of 1310 nm. Both the SFPs are made to
be compatible with each other.
12.3.6. Single-Mode Fibre, LC-LC (x2)
Figure 12.9: A 5m fibre cable by HP.
A fibre cable is needed for optical communication. We used HPE multimode OM4 2-fibre premier flex LC
to LC 5-meter cable (QK734A) 33 , which comes in a pair. We separated the fibre cables and used one of
them for our single-mode communication.

12.4. White Rabbit Switch

A switch is an essential part in any network, especially Gigabit Ethernet networks. Thus, we need a switch.
The White Rabbit Switch (WRS) is compatible with White Rabbit networks, and supports all WR features
allowing for synchronisation, determinism, and reliability.
12.4.1. The WRS
Figure 12.10: White Rabbit Switch (front and back).
WRS consists of 18 SFP ports, 5 SMC connectors, Xilinx Virtex-6 FPGA (LX240T FFG1156), ARM Atmel
AT91 SAM9G45 CPU, multiple oscillators and generators for clocking, and 2 cooling fans.
For management and diagnostics, WRS has Ethernet and UART management ports, and multiple LEDs to
indicate status.
It runs Linux kernel, and shell can be accessed via SSH. Switch can be configured via SSH or a web
management interface. We will discuss this later.
In default configuration, WRS acts as a boundary clock, with port 1 configured as a slave and the rest as
masters.
More information about the hardware can be found on the following page:
https://ohwr.org/project/wr-switch-hw/wikis/home.

12.4.2. RJ45 Patch-Cord
Figure 12.11: An RJ45 patch-cord.
An RJ45 patch-cord is needed to connect to the 100 Mbps Ethernet management port of WRS, which
allows us to assign an IP to WRS and connect to it via SSH.

13. Initial Setup
We will describe setups appropriately as we go along further in the project. Now that we are starting, we
first need to setup our system so that the cards are detected and drivers are installed properly.
We will set up
1. A point-to-point WR link, and
2. A network with WR switch.
Single mode Fiber Optic Cable
PC 1 PC 2
(Master) (Slave)
SFP SFP
SFP
SPEC SPEC
+ +
FMCDIO FMCDIO
Figure 13.1: A point-to-point WR link.

Ethernet
Management SFP SFP
White Rabbit Switch
Port (Master)
Single mode Single mode Fiber

Fiber Optic Optic Cable
RJ45 Cable
Patch
Cord
PC 1 PC 2
(Slave) (Slave)
SFP SFP
SFP
Ethernet
Port SPEC SPEC
+ +
FMCDIO FMCDIO
Figure 13.2: A WR network with WR switch.
The process to set everything up properly on a machine is as follows:
13.1. Hardware Installation

We first need to connect the FMC-DIO card to the SPEC board. After that, the SPEC board needs to be
connected to a PCIe slot in the host CPU.
Note that the DIO connectors will protrude out from the CPU.

Figure 13.3: A SPEC board (housing the FMC-DIO card) installed in a CPU.
13.2. Software Installation

Once the card has been installed in the CPU, boot up Linux (start Ubuntu LTS).
13.2.1. Detection of the SPEC Card

First check whether the boards have been successfully detected on our PCIe bus by doing the following:
1 $ lspci | grep CERN
If detected, the output should be as follows:
1 01:00.0 Non-VGA unclassified device: CERN/ECP/EDU Device 018d (rev 03)
If not detected, please make sure the card was installed correctly in the CPU.

13.2.2. Installation of Prerequisites

Before going with WR software installation, we need to install some prerequisite packages. The following
commands installs all of them:
1 $ sudo apt install git build-essential python-minimal wget libreadline-dev

linux-headers-$(uname -r) minicom texinfo emacs texlive-xetex pandoc iproute2
While not essential, it will be nice to have the latest cmake and ninja installed, since Ubuntu 16.04 has
old versions. We can do that by building from source (either clone as below, or get the latest release zip
from GitHub) as follows:
1 $ sudo apt remove cmake* ninja

2
3 $ git clone https://github.com/ninja-build/ninja.git && cd ninja
4 $ ./configure.py --bootstrap && sudo cp ninja /usr/local/bin
5 $ cd ../
6
7 $ git clone https://github.com/Kitware/CMake.git && cd CMake
8 $ ./bootstrap --parallel=`nproc` && make -j`nproc`
9 $ sudo make install
A latest version of Python is preferred. One can use Pyenv (https://github.com/pyenv/pyenv) to install new
versions locally. You should do it if you want to use pip behind a proxy, we have had problems regarding it
on Python 3.5.
13.2.3. Main Installation

Clone the White Rabbit Starting Kit repository from OHWR:
1 $ git clone https://ohwr.org/project/wr-starting-kit.git

2 $ cd wr-starting-kit && git submodule update --init --recursive && ls
We can see there is a Makefile for easy compilation. Thus, compile the kernel drivers and software tools by
executing make command as follows:
1 $ make -j`nproc`
(Note: If you want coloured output for better eyeballing, install colormake from GitHub
(pagekite/Colormake) and alias make to colormake .)

If the compilation succeeded, install the compiled drivers and tools by running the following command:
1 $ sudo make install
Finally, we have to load all the kernel drivers by executing the following commands one by one:
1 $ sudo modprobe spec

2 $ sudo modprobe htvic
3 $ sudo modprobe wr-nic
4 $ sudo modprobe wr-dio
(Note: Since we have to load the drivers every time we boot the system, we can automate the loading at
every boot by making a file /etc/modules-load.d/wr.conf and writing in it the driver/module names
(like spec ) each on a separate line.)
If all the steps have been followed correctly, the kernel logs ( dmesg ) will show the following:
1 [ 6.646851] spec 0000:01:00.0: probe for device 0001:0000

2 [ 6.646855] spec 0000:01:00.0: enabling device (0000 -> 0002)
3 [ 6.738167] spec 0000:01:00.0: got file "fmc/spec-init.bin", 1485236
(0x16a9b4) bytes
4 [ 6.929717] spec 0000:01:00.0: FPGA programming successful
5 [ 7.277672] spec 0000:01:00.0: mezzanine 0
6 [ 7.277673] Manufacturer: CERN
7 [ 7.277673] Product name: FmcDio5cha
8 [ 7.688425] spec 0000:01:00.0: got file "fmc/wr_nic_dio.bin", 1485512
(0x16aac8) bytes
9 [ 7.880007] spec 0000:01:00.0: FPGA programming successful
10 [ 8.457732] spec 0000:01:00.0: WRC program reloaded from
"fmc/wr_nic_lm32_sw.bin"
11 [ 14.373644] WR-nic: Using address 22:33:06:31:c2:bd
12 [ 14.373645] wr_nic: assign MAC 22:33:06:31:c2:bd to wri1
13 [ 14.373723] spec-nic spec-nic.0: White Rabbit NIC driver
14 [ 14.396857] DIO driver. Detected DIO V2
If you get stuck anywhere or want to know more, then you can read the detailed documentation by
building them yourself (run make in doc directory), or by downloading v3.1 docs from
https://ohwr.org/project/wr-starting-kit/wikis/Documents/version-'v3.1'-attachments.

14. Configuring and Synchronising the WR Nodes
We will be using two KIT SPEC cards — one will function as the master and other as the slave. Install the
cards in two different PCs by following the procedure mentioned in the previous section.
14.1. Virtual UART and Init Script

WR PTP Core (WRPC) running on an LM32 softcore processor has a virtual UART console which allows us
to configure the card and get diagnostic information over PCIe bridge, i.e., without the use of a physical
UART. We can access it using the wrpc-vuart command as follows:
1 sudo wrpc-vuart -f /sys/bus/pci/devices/0000:00:01.0/resource0 -o 0x20500
Whenever the card boots, it executes an initialisation ( init ) script. By default, the cards are configured as
slave. The init script, which can be seen using init show command, is as follows:
1 wrc# init show
For configuring a card as master, we need to issue the following commands on WRPC shell:
1 wrc# mode master

2 wrc# ptp start
In order to save ourselves from the trouble of manually executing the above commands, we can set the init
script to configure the card as master by issuing the following commands:
1 wrc# init add mode master

2 wrc# init add ptp start
The above appends the commands to the init script of card. One can also erase the init script entirely
using init erase command, but then one has to remember adding other appropriate commands from
default init script, like sfp match (which will be discussed below). To execute the script manually, one can
use the init boot command.

14.2. Master-Slave Connection

Now, we will connect the master and slave using a single-mode fibre cable. For that, first we need to plug
SFP (Small Form-factor Pluggable) optical transceivers into the two cards.
Figure 14.1: SFP slot location on board.
The violet SFP (an OLT transceiver) is to be plugged into the master, while the blue SFP (an ONU
transceiver) is to be plugged into the slave.
Before proceeding, we need to add calibration parameters for the SFPs in the SFP database of the card, so
that link delay is correctly calculated by White Rabbit. For getting the values, one has to do the White
Rabbit Calibration Procedure. 34
Since are using precompiled binaries available from OHWR, the values given at OHWR can be used
directly. 34 The values can be added using the sfp add command, as follows:
1 wrc# sfp add AXGE-1254-0531 180750 148326 72169888

2 wrc# sfp add AXGE-3454-0531 180750 148326 -73685416
Then, to load the appropriate parameters, one has to use the sfp match command. Now, we can simply
connect the single-mode LC-LC fibre cable to the two SFPs.
14.3. Synchronisation
Now that everything is set up, we can see the diagnostics and synchronisation information from wrpc shell
using the gui command, a screenshot of which on the slave side is below:

Figure 14.2: WRPC shell GUI on slave before syncing Linux system's time.
For slave to be completely synchronised to master, the servo state must be TRACK_PHASE . Sometimes due
to a bug in WRPC, it may get stuck at WAIT_OFFSET_STABLE when master goes down in between (note that
this is undesirable, hence a bug). In that case, disconnect and reconnect the fibre cable (one or multiple
times; typically once) till the TRACK_PHASE state is achieved.
Also, we can see that the time is not correct, it has started from the UNIX epoch time. To set the correct
current time, we can use the time setsec command at master.
Note that WRPC uses TAI time, and not UTC. TAI time is atomic time, while UTC time has had leap
seconds subtracted at multiple moments. Currently, TAI time is 37 seconds ahead of UTC, so while setting
time from Linux system clock, we need to add 37 seconds.
wrpc-vuart allows for giving wrpc shell commands using the -c option. Since we need to get current
system time, we can use this feature and Linux shell features to our advantage as follows:

1 sudo wrpc-vuart -f /sys/bus/pci/devices/0000:00:01.0/resource0 -o 0x20500 -c

"ptp stop"
"time setsec $(( $(date +%s) + 37 ))"
"ptp start"
(Note: While ptp stop and ptp start is not needed and time will be synced to the slave without it too,
we encountered some problems with FMC pulse generation (unable to generate with timestamp) when we
did not do it while distributing trigger later. LM32 softcore running the vuart shell is most probably at
fault here.)
Now the time will be set on master, and via White Rabbit PTP, the slave's time will also be synced. Thus,
checking again via gui command or just the time command, we can see the current time has been
synced.
Figure 14.3: WRPC shell GUI on slave after syncing Linux system's time.

15. Generation and Comparison of 1-PPS
15.1. Introduction
1-PPS (Pulse Per Second) signal is a good way to demonstrate synchronisation between two nodes and
checking the accuracy of White Rabbit network and the KIT SPEC card.
We will compare the 1-PPS generated by master with the 1-PPS generated by slave by checking the delay
between them using an oscilloscope.
15.2. Setup and Generation

Channel 0 (the first one) of the DIO on the board outputs a PPS signal by default. One can also configure
the channel to output DIO by using the wr-dio-cmd tool as follows (assuming the address of fmc-dio
device file as shown):
1 sudo wr-dio-cmd /dev/fmc-dio-1:0 mode 0 p
The above command should be run on both master and slave for completeness' sake. Then, a LEMO cable
should be used to connect the first channel on DIO side to an oscilloscope, as shown below:
Figure 15.1: A LEMO cable connected to channel 0 of the DIO.

The master should be connected to channel 1 of the oscilloscope, and the slave to channel 2. Trigger
should be set on rising edge of channel 1 (master) signal.
The time divisions should be set to nanoseconds level, like 5ns/div, since the (acceptable) delays will be at
ns level and thus would not be properly discernible if the time divisions are too big.
15.3. Observation
Figure 15.2: Train of 1-PPS signals by master visible on a 1s/div scale.

Figure 15.3: 1-PPS signals generated by master and slave on a 5ns/div scale.
Figure 15.4: The same 1-PPS signals as in the previous figure layered on top of each other.
We can see that they are almost overlapping. To get the separation, we use persistence mode of the
oscilloscope, so that many different samples can be seen, and zoom scale to the scope's maximum of
2ns/div.

Figure 15.5: Persistence display of 1-PPS signals by master and slave on a 2ns/div scale.
We can observe that the waveforms are at maximum separated by (cursor difference is shown on
the right), visibly showing us the sub-ns synchronisation.

16. Setting up White Rabbit Switch
White Rabbit Switch (WRS) v3.4 works out-of-the-box as there is a default configuration. But some
changes may be required. There are various config options which can be set by us. Various settings can be
changed via SSH or web interface. Also, WRS firmware should be upgraded to a newer one.
This section describes the process of doing the same.
16.1. Setting up a DHCP Server

A DHCP server is needed, because this server will assign IPs to the nodes, including the WRS. An IP needs
to be assigned to WRS because we need to SSH into WRS.
First, we need to install the DHCP daemon.
1 $ sudo apt install isc-dhcp-server
Then, we need to configure it, because we need to always assign the same IP to the Ethernet management
port on WRS based on MAC address, and also we can define our custom ranges for random assignment.
The configuration file is /etc/dhcp/dhcpd.conf . We set the contents to the following:
1 default-lease-time 600;
2 max-lease-time 7200;
3 authoritative;
4
5 # For random assignment.
6 subnet 192.168.5.0 netmask 255.255.255.0 {
7 range 192.168.5.100 192.168.5.250;
8 option routers 192.168.5.254;
9 }
10
11 # For WRS ethernet management port.
12 host archmachine {
13 hardware ethernet 00:1b:c5:09:00:a6;
14 fixed-address 192.168.5.1;
15 }

Now, restart the DHCP server with the following command.
1 $ sudo systemctl restart isc-dhcp-server.service
To check whether the service has been loaded successfully and the server is running, run the following
command:
1 $ sudo systemctl status isc-dhcp-server.service
16.2. Connecting WRS to DHCP Server

Use the RJ45 patch-cord to connect Ethernet management port of WRS (on the front) to the PC running
the DHCP server.
Power on the WRS. The status can be seen from WRS LEDs, and is described in WRS startup guide 35 as
follows:
1. On powering on, the Power LED goes green.
2. After 15 seconds, the Status LED goes orange which means that the WRS’ kernel has started.
3. Then the fans start spinning, which means that FPGA has been correctly programmed.
4. Finally, it goes green when everything is successful (PLL is locked).
Now test whether we can SSH into the WRS using the IP we have specified in dhcpd.conf . The login user
is root , and there is no password (simply press Enter when asked for it). We can connect via SSH using the
following command:
1 $ ssh root@192.168.5.1
Figure 16.1: Opening an SSH session on WRS.
To exit from an SSH session, press Ctrl+D, or type exit or logout .

16.3. Upgrading WRS Firmware

Latest version of WRS firmware is v6.0.2, while in the hardware we had v4.2. So we decided to upgrade.
The following commands can be used to upgrade the firmware from v4.1 36 and above:
1 $ wget https://ohwr.org/project/wr-switch-
sw/wikis/uploads/075468b4cd21a9bb8402bb9fdb186040/wr-switch-sw-v6.0.2-
20230224_binaries.tar
2
3 $ scp wr-switch-sw-v6.0.2-20230224_binaries.tar root@192.168.5.1:/update
After that, restart/reboot the switch (you can use reboot command via SSH).
16.4. Accessing the Web Management Interface

Since v6, the Web Management Interface is disabled by default due to vulnerabilities (CVE-2023-22577 is
specifically for WRS). 36
If the network is well-controlled, isolated, or air-gapped such that hacking remotely is not an issue, one
can enable it (though it is discouraged to do so).
To enable it, one needs to change the config options, which can be easily done in WRS via SSH.
Run the following command in an SSH session to get a TUI for changing config options:
1 $ wrs_menuconfig
Then, search for CONFIG_HTTPD_DISABLE , and press enter to remove the star inside the brackets in front of
the option. This sets it to no ( CONFIG_HTTPD_DISABLE=N ).
Save and exit, then reboot the WRS.
The web interface will now be enabled and can be accessed by pasting the IP assigned to it (here,
192.168.5.1) on a browser.

Figure 16.2: Web management interface.
16.5. Setting up the PHY for White Rabbit Network

Now since everything is at place, we can now connect WR nodes to it. In our case, we will connect the two
SPEC boards. The switch will act as the master, and the cards as the slave.
Connect two violet SFPs to ports 2 and 3, and connect the blue SFPs to each SPEC card. Then using single-
mode LC-LC fibre cables, connect the nodes to switch.
Make sure that the nodes are configures as slaves. On WRPC shell, use mode slave to configure the SPEC
board as a slave.

Then, use gui on WRPC shell to check calibration status. Both nodes should sync to WRS acting as a
master. The WRS status LEDs under the port should be blinking, indicating packet transfers (since PTP
information is continuously transmitted).
Figure 16.3: Active WRS ports.

17. Trigger Distribution Implementation
17.1. Introduction
Now that we have set up the requisite things, and ensured that the White Rabbit (WR) system is working
properly and the nodes are synchronised, we can now distribute triggers over the WR link.
As a first pass, a Python program was developed which made use of tools we get in WR Starting Kit, which
allowed us to easily check our setup practically and also run tests. But due to limitations of the tools, the
Python program could not get WRPC time in a fast way (the WRPC shell command in LM32 softcore
introduces a huge latency), and also it relied on polling by repeatedly using the CLI tools, as it had no
notion of interrupts (FMC-DIO generates an interrupt when a generation timestamp is added in an empty
FIFO, but we could not use it in Python).
It is pertinent to note that the tools are written in C, so with Python we are eventually interfacing with
some C code. So after that, a C program was written which directly interfaced with the kernel drivers.
FMC-DIO kernel driver was modified to return WRPC time it reads back to userspace.
We define trigger as a structure containing the following members:
1 struct trigger_data {
2 /* The FMC-DIO channel to generate trigger on. */
3 uint8_t channel;
4
5 /* Number of consecutive triggers to generate. */
6 uint32_t count;
7
8 /* WR TAI time of trigger generation. */
9 uint64_t when_sec;
10 uint32_t when_nsec;
11
12 /* Pulse width of trigger signal. */
13 uint64_t pulse_width_sec;
14 uint32_t pulse_width_nsec;
15
16 /* Period of trigger signal. */
17 uint64_t period_sec;
18 uint32_t period_nsec;
19 };

We used MQTT to distribute triggers from one node to another. MQTT was chosen because we can easily
define and mention node hierarchy, groups/sets, and endpoints by the use of topics.
The main intended trigger distribution program reads a list of triggers (with all the information and
associated node numbers) from a CSV file and distributes them using MQTT over WR network.
17.2. Setup
An MQTT broker needs to be running somewhere. We will run the broker at the node which will send the
triggers, i.e., the broker will run on the same PC as the publisher.
For the broker, we used nanomq. It is open-sourced on GitHub (emqx/nanomq), and both the source and
pre-compiled binaries can be downloaded from GitHub. If the pre-compiled binaries do not work, build the
source and install.
For communication over network to take place, we need to assign static IPs to the White Rabbit Interface
of the host that runs the MQTT broker. The interface is typically named as wriX (where X is a number; for
example, wri1 ). The interface can be found using the ip command ( ip addr ). Then, assign IP addresses
to this interface. You can either do that via GUI by going into settings, or with the help of command line as
follows:
1 $ sudo ip addr add 10.0.0.1/24 dev wri1
The above command will set the IP of wri1 interface as 10.0.0.1/24 . This can be verified as follows:
1 $ ip addr show dev wri1

2 3: wri1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UNKNOWN group default qlen 1000
3 link/ether 22:33:06:31:c2:bd brd ff:ff:ff:ff:ff:ff
4 inet 10.0.0.1/24 brd 10.0.0.255 scope global wri1
5 valid_lft forever preferred_lft forever
6 inet6 fe80::9072:e867:d912:f921/64 scope link
7 valid_lft forever preferred_lft forever
This IP can now be used for connecting to the MQTT broker. The full trigger distribution flow is as follows:
1. Start the broker (in our case, using nanomq start command).
2. Run the subscriber program at the nodes, which will connect to the broker (using its IP)
specifying topics and LWT, and listen for triggers continuously.
3. Run the publisher program which will send the triggers over the MQTT network and finish
executing.

17.3. Determining Resolution

Since triggers will be eventually generated by giving commands to the FMC-DIO, the resolution of the
trigger distribution system will be the resolution of the DIO.
To determine the resolution, we will use the wr-dio-cmd tool, which allows us to generate pulses by
specifying time (absolute or relative), pulse width, period, and count. We will issue commands to generate
multiple pulses with differing pulse width. The time (difference) at which the pulse generates or
differentiates from the previous pulse width will be our resolution.
Since we have sub-ns accuracy, good shot is to try deltas like 1ns, 2ns, 3ns, etc. Execute the tool using the
terminal as follows:
1 # 1 ns pulse width.
2 sudo wr-dio-cmd /dev/fmc-dio-1:0 pulse 1 0.0000000001 now
3
6
9
10 # and continue like that.
17.4. Performance Testing

One node will send trigger to another node. We will do performance testing with our two network setups,
viz., point-to-point (a direct link between master and slave cards, with master sending the trigger to
slave), and switched (a WR switch is the master, with the other two nodes being slave).
In this way, we can reason about network performance and the impact of WR switch on it.
17.4.1. Point-To-Point
17.4.1.1. Single Trigger Distribution Test
We will first send one trigger at a given / predefined time. The trigger distribution program is modified
such that the master (which will send the trigger) will also generate the trigger, so that we can compare
the trigger waveform alongside that of slave's and check if there is any significant jitter.

We will set a trigger level and set the trigger source to channel 1 of the oscilloscope. When the triggers are
generated, cursors can be used to get the separation between the waveforms. Since the sample shown is
because the voltage at channel 1 crossed the set trigger level, we have a reference. Thus, the two X cursors
have to be set at the points where the two waveforms cross the trigger level line.
17.4.1.2. Multiple Trigger Distribution Test
We now know that upon sending a single trigger, we don't see any significant jitter. But we only saw for
the particular trigger we sent at that time, and that does not give us any upper bound. Also, what about
multiple triggers? Will there be any significant jitter if multiple triggers are sent or if the system works for
a prolonged period of time? Thus, we need to stress test the system.
Thus, we will send multiple triggers in quick succession (pulse width = 5 ms, period = 10 ms, time between
triggers = 100 ms).
Whenever a trigger is generated on a particular channel, the timestamp of generation is saved in a FIFO
queue associated with the channel in the hardware. The test program saved all the timestamps of trigger
generation at both the master and slave during runtime.
Also, the oscilloscope was set in persistent mode so that we can see the maximum amount of deviation /
jitter, since we cannot humanly see all the signals and check the jitter in real time. In persistent mode, the
oscilloscope will layer each signal on trigger on top of each other. Thus, we can later verify the maximum
deviation.
17.4.2. Network with WR Switch
17.4.2.1. Single Trigger Distribution Test
We use the same setup (test program, oscilloscope) as in the point-to-point case. The only difference is
that now our SPEC cards are connected as slaves to a switch acting as a master.
17.4.2.2. Multiple Trigger Distribution Test
We use the same setup (test program, oscilloscope, trigger generation frequency) as in the point-to-point
case. The only difference is that now our SPEC cards are connected as slaves to a switch acting as a master.

18. Observation and Results
18.1. Resolution
Table 18.1: Pulse width issued in command v/s pulse width actually generated, as seen on the oscilloscope.
PULSE WIDTH ISSUED IN COMMAND PULSE WIDTH GENERATED ACTUALLY
1 ns No pulse generated.
8 ns 8 ns
9 ns 8 ns
10 ns 8 ns
11 ns 8 ns
12 ns 8 ns
13 ns 8 ns
14 ns 8 ns
15 ns 8 ns
16 ns 16 ns
17 ns 16 ns
18 ns 16 ns
19 ns 16 ns
20 ns 16 ns
21 ns 16 ns
22 ns 16 ns
23 ns 16 ns
24 ns 24 ns

PULSE WIDTH ISSUED IN COMMAND PULSE WIDTH GENERATED ACTUALLY
25 ns 24 ns
26 ns 24 ns
We can observe that the pulse width changes every 8 ns. Thus, the resolution is 8ns. This is to be
expected, as the default clock period of WRPC in the SPEC card is 8 ns (125 MHz clock). Thus, the time and
time periods in all the time related fields in the struct trigger_data must be a multiple of the clock
period, here 8 ns.
Note that this does not mean that the White Rabbit network itself is limited to 8 ns resolution. White
Rabbit provides us with sub-ns accuracy and picosecond-level precision. If the frequency at the White
Rabbit node (here, SPEC board) was higher, the clock period would be smaller, and hence the resolution
would be finer.
18.2. Point-To-Point
18.2.1. Single Trigger Distribution Test

No significant jitter was observed for the single trigger we sent. We get sub-nanosecond accuracy, and this
trigger happens to have jitter, as can be seen in figure 18.2.
Figure 18.1: Point-to-point trigger test's two trigger waveforms on a 10ms/div scale.

Figure 18.2: The two trigger waveforms on a 2ns/div scale. Cursors are used to get the separation ( in the
right box).
18.2.2. Multiple Trigger Distribution Test

The test was run for 30 minutes.
It was observed from the CSV file (which recorded all the trigger generation timestamps at master and
slave) that both the master and slave columns were the same (the sequence of generation timestamps
were same at both master and slave), as expected.

Figure 18.3: Oscilloscope display at the end of point-to-point multiple trigger distribution test.
The Y-axis cursors are set at mark for the respective triggers (set equally above the 0 level in both
channels, with the trigger level coinciding). The X-axis cursors are then set to where the furthest trace
intersects the respective Y-cursor. As can be observed, the jitter lies in a band. The maximum jitter is
.
We can definitely say it is maximum (unless there is some edge case we did not encounter in 30 minutes).
Actual might be lower, because due to persistence, we overlay all the signals, but that does not necessarily
mean the furthest traces happened simultaneously, i.e., it is unlikely that the leftmost trace at channel 1
and the rightmost trace at channel 2 occurred at the same time. A visual observation of the process shows
that majorly it is not the case.
18.3. Network with WR Switch
18.3.1. Single Trigger Distribution Test

No significant jitter was observed for the single trigger we sent. We get sub-nanosecond accuracy, and this
trigger happens to have jitter.

Figure 18.4: Switched single trigger test's waveforms on a 2ns/div scale.
18.3.2. Multiple Trigger Distribution Test

The test was run for 30 minutes.
It was observed from the CSV file (which recorded all the trigger generation timestamps at master and
slave) that both the master and slave columns were the same (the sequence of generation timestamps
were same at both master and slave), as expected.
As can be observed in Figure 18.6 and using the previously mentioned arguments, the maximum jitter is
.
We can see that the jitter has increased after adding the switch to the network and using that as the
master.

Figure 18.5: Oscilloscope display at the end of switched multiple trigger distribution test.

19. Conclusion
White Rabbit (WR) provides a promising feature set for deploying trigger distribution systems in
distributed systems having hard-real time requirements. White Rabbit provides us with sub-nanosecond
accuracy and picoseconds level of synchronisation (precision) over a gigabit Ethernet network with optical
fibres. PTP is used for synchronisation, and SyncE is used for syntonisation, so that the accuracy of
synchronisation increases, and the problem of link delay is converted to a problem of phase difference.
DDMTD is used for phase correction with sub-picosecond accuracy, and we achieve picosecond precision
for WR timestamps.
We elucidated upon the main White Rabbit concepts and associated things, and also explained the
apparatus and gatewares used. We then proceeded to design a custom trigger distribution utility which
runs over WR network using MQTT. We have shown that for the KIT SPEC card in the White Rabbit
starting kit, we get sub-ns jitter, and a resolution of 8 ns, which was due to SPEC card's default clock speed
being 125 MHz.
By deploying a WR network, cabling will simplify as data and timing are transmitted simultaneously over a
single PHY. Thus, separate timing and communication networks in systems like SST-1 can be replaced
with a single WR network. This will simplify the technical debt and lower down the costs, while getting
better performance. Thus, by Occam's razor, the feasibility of deployment is justified.
Future avenues can include adding support for trigger distribution directly in the gateware (i.e., in HDL
code), and subsequently implementing both trigger distribution and data acquisition systems
simultaneously.

References
[1] Institute for Plasma Research, "History." ipr.res.in. http://www.ipr.res.in/documents/about_us.html

(accessed May 3, 2023). Archived: https://web.archive.org/web/20230503110804/https://www.ipr.res.in/do
cuments/about_us.html. ↩ ↩
[2] M. Lipinski, "White Rabbit Official CERN Wesbite." white-rabbit.web.cern.ch. https://white-rabbit.web.c

ern.ch (accessed May 1, 2023). Archived: https://web.archive.org/web/20230501105045/https://white-rabb
it.web.cern.ch/. ↩
[3] J. Serrano, M. Cattin, E. Gousiou, E. van der Bij, T. Włostowski, G. Daniluk, M. Lipiński, "The White
Rabbit Project" in proceedings of IBIC2013, Oxford, UK, pp. 936-942. Available: https://accelconf.web.cern.c
h/ibic2013/papers/thbl2.pdf (accessed May 1, 2023). Archived: https://web.archive.org/web/202208190715
34/https://accelconf.web.cern.ch/IBIC2013/papers/thbl2.pdf. ↩ ↩ ↩
[4] P.P.M. Jansweijer, H.Z. Peek, E. De Wolf, "White Rabbit: Sub-Nanosecond Timing over Ethernet", Nucl.
Instrum. Methods Phys. Res. A, 725, pp. 187-190, Oct 2013. Available: https://indico.cern.ch/event/143656/
papers/1378076/files/1031-WHITE.pdf (accessed May 1, 2023). Archived: https://web.archive.org/web/202
30501093819/https://indico.cern.ch/event/143656/papers/1378076/files/1031-WHITE.pdf. ↩
[5] "IEEE Standard for Ethernet," in IEEE Std 802.3-2022 (Revision of IEEE Std 802.3-2018), pp.1-7025, 29
July 2022, doi: 10.1109/IEEESTD.2022.9844436. Available: https://ieeexplore.ieee.org/document/9844436
(accessed May 1, 2023). ↩ ↩ ↩ ↩
[6] Milijevic, Slobodan. "The basics of synchronized Ethernet." EE Times-Asia, 2009. Available: https://ww
w.microsemi.com/document-portal/doc_view/126475-the-basics-of-synchronized-ethernet-synce
(accessed May 1, 2023). Archived: https://web.archive.org/web/20230501094653/https://www.microsemi.c
om/document-portal/doc_view/126475-the-basics-of-synchronized-ethernet-synce. ↩ ↩ ↩ ↩
[7] "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and
Control Systems," in IEEE Std 1588-2019 (Revision of IEEE Std 1588-2008), pp.1-499, 16 June 2020, doi:
10.1109/IEEESTD.2020.9120376. Available: https://ieeexplore.ieee.org/document/9120376 (accessed May
1, 2023). ↩ ↩
[8] "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and
Control Systems," in IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002), pp.1-269, 24 July 2008, doi:
10.1109/IEEESTD.2008.4579760. Available: https://ieeexplore.ieee.org/document/4579760 (accessed May
1, 2023). ↩

[9] M. Lipinski "White Rabbit integration into IEEE1588-2019 as High Accuracy" ohwr.org. https://ohwr.org/p
rojects/wr-std/wiki/wrin1588 (accessed May 1, 2023, commit 98fac7b8). Archived: https://web.archive.org/
web/20230501111457/https://ohwr.org/projects/wr-std/wiki/wrin1588. ↩
[10] M. Lipiński, T. Włostowski, J. Serrano and P. Alvarez, "White rabbit: a PTP application for robust sub-
nanosecond synchronization," 2011 IEEE International Symposium on Precision Clock Synchronization for
Measurement, Control and Communication, Munich, Germany, 2011, pp. 25-30, doi:
10.1109/ISPCS.2011.6070148. Available: https://white-rabbit.web.cern.ch/documents/White_Rabbit-a_PTP
_application_for_robust_sub-nanosecond_synchronization.pdf (accessed May 1, 2023). Archived: https://w
eb.archive.org/web/20230501111624/https://white-rabbit.web.cern.ch/documents/White_Rabbit-a_PTP_ap
plication_for_robust_sub-nanosecond_synchronization.pdf. ↩ ↩ ↩
[11] P. Moreira, P. Alvarez, J. Serrano, I. Darwezeh and T. Wlostowski, "Digital dual mixer time difference
for sub-nanosecond time synchronization in Ethernet," 2010 IEEE International Frequency Control
Symposium, Newport Beach, CA, USA, 2010, pp. 449-453, doi: 10.1109/FREQ.2010.5556289. Available: http
s://white-rabbit.web.cern.ch/documents/DDMTD_for_Sub-ns_Synchronization.pdf (accessed May 1, 2023).
Archived: https://web.archive.org/web/20230501111837/https://white-rabbit.web.cern.ch/documents/DD
MTD_for_Sub-ns_Synchronization.pdf. ↩ ↩
[12] Timing and synchronization aspects in packet networks, Recommendation ITU-T G.8261/Y.1361, Aug
2019. Available: https://www.itu.int/rec/T-REC-G.8261-201908-I/en (accessed May 1, 2023). Archived: htt
ps://web.archive.org/web/20230501094939/https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-G.
8261-201908-I%21%21PDF-E&type=items. ↩
[13] Timing characteristics of synchronous equipment slave clock, Recommendation ITU-T G.8262/Y.1362,
Nov 2018. Available: https://www.itu.int/rec/T-REC-G.8262-201811-I/en (accessed May 1, 2023). Archived:
https://web.archive.org/web/20230501095205/https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC
-G.8262-201811-I%21%21PDF-E&type=items. ↩ ↩
[14] Distribution of timing information through packet networks, Recommendation ITU-T G.8264/Y.1364,
Aug 2017. Available: https://www.itu.int/rec/T-REC-G.8264-201708-I/en (accessed May 1, 2023). Archived:
https://web.archive.org/web/20230501095214/https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC
-G.8264-201708-I%21%21PDF-E&type=items. ↩
[15] R. Razzaghi, A. Derviskadic and M. Paolone, "A white rabbit synchronized PMU," 2017 IEEE PES
Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), Turin, Italy, 2017, pp. 1-6, doi:
10.1109/ISGTEurope.2017.8260178. ↩
[16] "IEEE Standard for Local and Metropolitan Area Networks--Bridges and Bridged Networks," in IEEE
Std 802.1Q-2022 (Revision of IEEE Std 802.1Q-2018), pp.1-2163, 22 Dec. 2022, doi:
10.1109/IEEESTD.2022.10004498. Available: https://ieeexplore.ieee.org/document/10004498 (accessed
May 1, 2023). ↩ ↩
[17] M. Lipinski, "Methods to Increase Reliability And Ensure Determinism in a White Rabbit Network,"
Ph.D. dissertation, Faculty of Electronics and Information Technology, Warsaw U. of Tech., Warsaw, 2016.

Available: https://cds.cern.ch/record/2261452 (accessed May 1, 2023). Archived: https://web.archive.org/w

eb/20230501113935/https://cds.cern.ch/record/2261452/files/CERN-THESIS-2016-283_2.pdf. ↩ ↩
[18] D. Akula, "What is a Distributed System?." geeksforgeeks.org. https://www.geeksforgeeks.org/what-is-a

-distributed-system/ (accessed May 1, 2023). Archived: https://web.archive.org/web/20230501091502/http
s://www.geeksforgeeks.org/what-is-a-distributed-system/. ↩
[19] Institute for Plasma Research, "Steady State Superconducting Tokamak (SST-1)." ipr.res.in. https://ww
w.ipr.res.in/documents/sst-1.html accessed May 3, 2023). Archived: https://web.archive.org/web/20230509
071342/https://www.ipr.res.in/documents/sst-1.html. ↩
[20] G. Daniluk, "White Rabbit PTP Core the sub-nanosecond time synchronization over Ethernet," M.S. thesis,
Faculty of Electronics and Information Technologies, Warsaw U. of Tech., Warsaw, 2012. Available: https://
ohwr.org/project/white-rabbit/uploads/479c4673c48eae8fe17816d144921217/GD_mgr.pdf (accessed May
1, 2023). Archived: https://web.archive.org/web/20230501095708/https://ohwr.org/project/white-rabbit/u
ploads/479c4673c48eae8fe17816d144921217/GD_mgr.pdf. ↩ ↩
[21] J. Díaz, M. Jiménez, R. Rodriguez, B. Rat, "White-Rabbit NIC Gateware." ohwr.org. https://ohwr.org/pro
ject/wr-nic/uploads/059ec8debbba5a47338364d7c30bc1aa/wr-nic-v2.0-170214.pdf (accessed May 1, 2023).
Archived: https://web.archive.org/web/20230501102358/https://ohwr.org/project/wr-nic/uploads/059ec8d
ebbba5a47338364d7c30bc1aa/wr-nic-v2.0-170214.pdf. ↩ ↩ ↩
[22] M. Lipinski et. al, "Wr streamers." ohwr.org. https://ohwr.org/project/wr-cores/wikis/WR-Streamers

(accessed May 9, 2023, commit 09732469). Archived: https://web.archive.org/web/20230509050959/http
s://ohwr.org/project/wr-cores/wikis/WR-Streamers. ↩ ↩ ↩
[23] W. Terpstra, "EtherBone Core Wiki." ohwr.org https://ohwr.org/project/etherbone-core/wikis/home

(accessed May 9, 2023, commit 56e6030a). Archived: https://web.archive.org/web/20230509055524/http
s://ohwr.org/project/etherbone-core/wikis/home. ↩ ↩
[24] A. Rubini, J. M. Cano, "FMC DIO Software Support." ohwr.org. https://ohwr.org/project/fmc-dio-5chttl

a/blob/master/doc/fmc-dio.in (accessed May 1, 2023, commit b0105d4f). Archived: https://web.archive.or
g/web/20230501102706/https://ohwr.org/project/fmc-dio-5chttla/raw/master/doc/fmc-dio.in. ↩
[25] A. Banks, E. Briggs, K. Borgendale, R. Gupta, "MQTT Version 5.0," OASIS Standard, 07 March 2019.
Available: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html (accessed May 1, 2023).
Archived: https://web.archive.org/web/20230501110652/https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/m
qtt-v5.0-os.html. ↩ ↩ ↩ ↩
[26] Keysight Technologies, "DSOX3034A Oscilloscope: 350 MHz, 4 Channels | Keysight." keysight.com. http
s://www.keysight.com/in/en/product/DSOX3034A/oscilloscope-350-mhz-4-channels.html (accessed May
1, 2023). Archived: https://web.archive.org/web/20230501105827/https://www.keysight.com/in/en/produc
t/DSOX3034A/oscilloscope-350-mhz-4-channels.html?&cc=US&lc=eng. ↩

[27] Seven Solutions, "WR Starting Kit." ohwr.org. https://ohwr.org/project/wr-starting-kit/wikis/uploads/e

97b42b1911c91d18c3d4631b1c12de2/wr-starting-kit.pdf (accessed May 1, 2023). Archived: https://web.arc
hive.org/web/20230501110057/https://ohwr.org/project/wr-starting-kit/wikis/uploads/e97b42b1911c91d1
8c3d4631b1c12de2/wr-starting-kit.pdf. ↩
[28] Seven Solutions, "KIT SPEC » Seven Solutions," sevensols.com. https://sevensols.com/kit-spec/

(accessed May 1, 2023). Archived: https://web.archive.org/web/20230501110109/https://sevensols.com/kit
-spec/. ↩
[29] Creotech, "Simple PCIe FMC Carrier - SPEC - Creotech," creotech.pl. https://creotech.pl/product/simpl
e-pcie-fmc-carrier-spec/ (accessed May 1, 2023). Archived: https://web.archive.org/web/20230501110224/
https://creotech.pl/product/simple-pcie-fmc-carrier-spec/. ↩
[30] Creotech, "FMC DIO 5CH TTL A - Creotech," creotech.pl. https://creotech.pl/product/fmc-dio-5ch-ttl-

a/ (accessed May 1 2023). Archived: https://web.archive.org/save/https://creotech.pl/product/fmc-dio-5ch-
ttl-a/. ↩
[31] Axcen Photonics Corporation, "1.25Gbps Single Fiber Bi-directional SFP, OLT Transceiver," AXGE-
3454-0531 datasheet rev. 1.5, Aug 2019. Available: https://www.axcen.com.tw/upload/product/202004281
726080.pdf. Archived: https://web.archive.org/web/20230501110204/https://www.axcen.com.tw/upload/pr
oduct/202004281726080.pdf. ↩
[32] Axcen Photonics Corporation, "1.25Gbps Single Fiber Bi-directional SFP, ONU Transceiver," AXGE-
1254-0531 datasheet rev. 1.5, Aug 2019. Available: https://www.axcen.com.tw/upload/product/202004281
724140.pdf. Archived: https://web.archive.org/web/20230501110233/https://www.axcen.com.tw/upload/pr
oduct/202004281724140.pdf. ↩
[33] Hewlett Packard Enterprises, "HPE multimode OM4 2-fibre premier flex LC to LC 5-meter cable,"
QK734A datasheet ver. 5, Oct 2022. Available: https://www.hpe.com/psnow/doc/c04123143.html?jumpid=i
n_pdp-psnow-qs&ver=5 (accessed May 1, 2023). Archived: https://web.archive.org/web/20230501110437/h
ttps://www.hpe.com/psnow/doc/c04123143.html?jumpid=in_pdp-psnow-qs&ver=5. ↩
[34] G. Daniluk, et. al, "White Rabbit devices calibration." ohwr.org. https://ohwr.org/projects/white-rabbit/
wiki/Calibration (accessed May 1, 2023, commit cd423190). Archived: https://web.archive.org/web/202305
01105348/https://ohwr.org/projects/white-rabbit/wiki/Calibration. ↩ ↩
[35] B. Rat, R. Agis. "WRS-3/18 - Startup Guide." ohwr.org. https://ohwr.org/project/wr-switch-sw/tree/wr-s

witch-sw-v6.0.2/doc/startup-guide (built May 1, 2023, hash 4d469ca5). ↩
[36] J-C. Bau, G. Daniluk, M. Lipinski, B. Rat, A. Rubini, F. Vaga, A. Wujek. "White Rabbit Switch: User’s
Manual." ohwr.org. https://ohwr.org/project/wr-switch-sw/wikis/uploads/474616590d196e97cd8b7f933117
40c7/wrs-user-manual-v6.0.2.pdf (accessed May 1, 2023). Archived: https://web.archive.org/web/20230501
112941/https://ohwr.org/project/wr-switch-sw/wikis/uploads/474616590d196e97cd8b7f93311740c7/wrs-u
ser-manual-v6.0.2.pdf. ↩ ↩

Study of White Rabbit Based Real-Time Ethernet For Distributed Control System

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Study of White Rabbit Based Real-Time Ethernet For Distributed Control System

Uploaded by

Copyright:

Available Formats

STUDY OF WHITE RABBIT BASED REAL-TIME

ETHERNET FOR DISTRIBUTED CONTROL SYSTEM

SIDDH RAMAN PANT (190170111075)

In partial fulfillment for the award of the degree of:

Gujarat Technological University, Ahmedabad

Date: 10th May, 2023

Prof. Shreeji H. Shah Dr. Arun B. Nandurbarkar

Assistant Professor, EC Head of the Department, EC

We hereby declare that

Date: 10th May, 2023

Siddh Raman Pant Akhilesh Bahuguna

– Siddh Raman Pant and Akhilesh Bahuguna

NUMBER CAPTION PAGE

Figure 4.3 Full PTP synchronisation flow. 9

Figure 12.8 AXGE-1254-0531 and AXGE-3454-0531 SFP transcievers. 37

Figure 15.5 Persistence display of 1-PPS signals by master and slave on a 53

Figure 16.2 Web management interface. 57

Figure 18.1 Point-to-point trigger test's two trigger waveforms on a 10ms/div 64

Figure 18.4 Switched single trigger test's waveforms on a 2ns/div scale. 67

NUMBER TABLE PAGE

Table of Contents vii

1.3. DACD Lab at IPR 2

4. Precision Time Protocol 7

5.3. SyncE in White Rabbit 12

7. Virtual Local Area Network 15

7.3. Data Communication in White Rabbit 16

9.2. Need for Trigger Distribution 20

10. White Rabbit Gatewares 22

10.1. White Rabbit PTP Core (WRPC) 22

10.3.1. Plain Fabric 24

10.5. An Approach for Trigger Distribution at FPGA Level 27

11.3.1. Quality of Service 30

11.3.2. Control Messages 31

11.3.3. Last Will and Testament 32

12.1 Desktops with Linux Kernel v4.x 33

12.2. Digital Oscilloscope with Nanoseconds Level Zoom 34

12.3. White Rabbit Starting Kit 34

12.3.2. FMC Digital I/O, 5 Channel, TTL A (x2) 36

12.3.3. LEMO 00 Connector (x3) 36

12.3.4. LEMO-BNC Adaptor (x3) 37

12.3.6. Single-Mode Fibre, LC-LC (x2) 38

12.4. White Rabbit Switch 39

12.4.1. The WRS 39

13. Initial Setup 41

13.1. Hardware Installation 42

13.2. Software Installation 43

13.2.2. Installation of Prerequisites 44

14.1. Virtual UART and Init Script 46

15. Generation and Comparison of 1-PPS 50

16.2. Connecting WRS to DHCP Server 55

16.5. Setting up the PHY for White Rabbit Network 57

17.4. Performance Testing 61

17.4.1.1. Single Trigger Distribution Test 61

17.4.2.1. Single Trigger Distribution Test 62

18.2.1. Single Trigger Distribution Test 64

18.3.1. Single Trigger Distribution Test 66

18.3.2. Multiple Trigger Distribution Test 67

ㅤ VGEC, Chandkheda Page 1 of 73

1.3. DACD Lab at IPR

Figure 1.1: New Research and Development Laboratory Building at IPR.

ㅤ VGEC, Chandkheda Page 2 of 73

2. Introduction to White Rabbit

ㅤ VGEC, Chandkheda Page 3 of 73