DCM Embedded Software

Infrastructure, Build Environment

and Kernel Modules

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 1

 What is theDCM Embedded System?
• The OS & specialized drivers/software required to talk to and extract
information from the hardware.
• Specifically:
– Configuration/Boot Method (flash memory + bootloader)
– Limited Operating System
• Includes Kernel & specialized device drivers
• A memory based filesystem
• Configuration parameters
• Limited set of command utilities
• Small enough to fit entirely in memory
– Application code
• Designed (and optimized) specifically for the hardware
• Compiled and run strictly out of device memory (small footprint)
• Designed for “diskless” I/O

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 2

 DCM Embedded System
DAQ Server
Boot
DCM
Loader
Mem (512mb)
Non-Volatile
Mem Kernel
Hardware
DCM
Registers Kernel Module Initial RamDisk
Provides:
/dev/dcm

Available Root File System

Configuration (RamDisk)
Memory
Sandbox
Links (477mb) NSF Mounts

For App + Buff NSF Mounts

NSF Mounts

DCM Can now run application code

and access hardware buffers from
user space 3
 DCM Boot Example:
• Behold a working DCM: (switch to example)

• Boot/Configure time:
– Average 38sec
– Includes ssh initialization for diagnostics
– Does NOT include startup of user applications
– No calls to external DBs etc…
• This is on NOvA test stand setup in Feynman
– Quiet network with dedicated server
A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 4
 DCM Kernel Driver
• Stat • D
• Currently 1 module us at
(dcm.ko) Buff a
• Contains correct • Contr B
er
microcode and register ol Reg u
map for current DCM • Kernel •
ffSingle Word
Transfer
prototypes • dcm
er
• Provides access to the .ko
• User Space
control registers Programs
• Provides data readout in (i.e.
single word reads
dcm_contr
A.Norman (U.Virginia) ol)
July '09 NOvA Collaboration Mtg. 5
 Kernel: Not Implemented•
• Stat D
• No DMA transfer
Empty Buffer
us at
– Partially implimented Buff a
– Does provide block • Contr B
er
transfers ol Reg u
– BUT….results in data
corruption and repeated • Kernel •
ffDMA Block
Transfer
• dcm
data blocks er
• Status block is defined • User Space .ko
but not implemented Programs Data Corruption
• NOvA Event View in the (i.e. & Repeated Events
works for Diagnostics
dcm_contr
A.Norman (U.Virginia) ol)
July '09 NOvA Collaboration Mtg. 6
 FEB Integration
• With the current kernel module:
– Can load the control registers
– Can read the control registers
FEB/DCM Integration Test
• This means we can technically push data to a DCM
Scheduled for Next Week
which is tagged to be transmitted to the FEB.
We’ll know at this point how
– This interface is going to change slightly, but the format is
just a memory mapped I/O
this works in practice
DCM Command FEB Address Data Word for FEB

Command processed by DCM Data required for given FEB command

(i.e. WRITE_TO_FEB or
READ_FROM_FEB)
Address in FEB of FEB command register
7
 Designing for 180+ Systems
• Design:
– Want lightweight identical embedded systems
– Need 180+ simultaneous copies running
• Problem:
– Each DCM system needs to be identical, but each
needs to have a unique identity
“You are a unique DCM, just like everybody else”
– Means having a method of passing identity
information at startup and embedding it in the system
Having a DCM “Know who it is” simplifies Run Control and
Resource Manager Operations
A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 8
 DCM Idenity Encode a 16bit word
with the DCM position
DCHP Table 0x0Bxx0Ayy
DHCP Request
Database
Keys off device MAC DHCP Response DCM oUniversal
DCM Kernel Module
DCM ID
Static Assignment of IP: 192.168.x.x MAC: 00-FF-FF….FF oDCMKernel
MACIdent.
IP address at boot oInstallWord:
Position
TFTP Config
Assigns config Assignment 0x0Bxx0Ayy
images This info comes
from installer

Every DCM is forced to “Know who it is”

TFTP Config
Request Config Load
order to take data.
(Kernel/RAMDISK) This is independent from Run Control
Detector or Block
a Resource Manager
Configuration Sandbox 1-15
Kernels RamDisk Images
Kernels 180+ configurations
RamDisk Images
Kernels
Kernels RamDiskImages
RamDisk Images
Kernels RamDisk Images
Kernels RamDisk Images 9
 Dev. & Build Environment
• DCM is a PowerPC based system
– Requires cross-compilation ofSimple
all applications
Cross Compilation
– Cross-compilers Flow Example
are in place and working
Build Machine
• Both a(Intel
kernel compiler
i686 Arch) and app compilerTarget
are working
Machine
• Simple (Motorola PPC 603 Arch)
code build system works for custom applications
– Need to bootstrap
Compiler newer compilers to accommodate external
Foreign
(cross Arch)
software Libraries
packages chosen by the DAQ systemPPC(example:
Libraries
Ganglia package
Obj Code
(PPC form)
for DAQMon) (Local Copy)

– Build examples exist for correct cross compiles of most

Linker Executable Executable
applications
(cross Arch) (Foreign) (PPC)
• But this needs to be formalized to match the rest of the DAQ build
Copy across
model network

10
 In the works
Working Still Required
• Bootloader (via debug trig) • Boot on power up
• DCM base OS and config. • OS Optimizations
• DCM Kernel Module • DCM Kernel Module
– Data Buffer
– Control Reg
(DMA block mode)
– Data Buffer – Status Buffer
(single word mode) – FEB programming
– User space hooks • FEB Kernel Module
• One catch all User app. – Simplify FEB configurations
– dcm_control • User apps
• Read/write – Need diagnostics
• FPGA loading • Memory views, register views, etc…
• Etc… • Expert applications for debugging

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 11

 End

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 12

 Status Status
Buffer Buffer Data
Data
Control Reg Buffer Control Reg Buffer

Kernel Single Word Kernel DMA Block

Transfer Transfer
dcm.ko dcm.ko

User Space Programs User Space Programs

(i.e. dcm_control) (i.e. dcm_control)

Data Corruption
& Repeated Events

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 13

 Simple Cross Compilation
Flow Example
Build Machine
(Intel i686 Arch) Target Machine
(Motorola PPC 603 Arch)
code
Compiler Foreign
(cross Arch) Libraries PPC
Obj Code Libraries
(PPC form) (Local Copy)

Linker Executable Executable

(cross Arch) (Foreign) (PPC)
Copy across
network

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 14

 Outline of what to cover
• Overview
• Hardware/Software diagram
• Hardware configuration and loading
• Initial boots
• RAM disks

A.Norman (U.Virginia) July '09 NOvA Collaboration Mtg. 15