IX Workshop on Real-Time Systems

39

MIMO - A Hybrid Monitor for Embedded Real-Time Systems

Luiz F. Copetti, Joo Cadamuro Jr., Douglas P. B. Renaux, Volnei A. Pedroni Programa de Ps-Graduao em Engenharia Eltrica e Informtica Industrial Universidade Tecnolgica Federal do Paran (UTFPR) Av. Sete de Setembro, 3165 80.230-901 Curitiba PR Brazil
copetti@utfpr.edu.br, joao.cadamuro@lactec.org.br, douglas@lit.citec.cefetpr.br, pedroni@utfpr.edu.br

Abstract. Testing embedded real-time systems with hybrid monitors is an interesting approach because these monitors offer the accuracy and low intrusion of hardware-only monitors and the flexibility of software-only monitors. However, the hybrid monitor itself is an embedded real-time system with its own requirements, including efficiency and flexibility. In this paper we present the design principles and the performance considerations for the MIMO (Minimal Invasive Monitor), a hybrid monitor used in academic research work.

1. Introduction
After implementation, an embedded real-time system should be verified against both logical and temporal requirements. The use of formal methods at design time may reduce the number of temporal faults, but even formal methods do not detract from the need for testing. They are complementary strategies [Burns 1997]. Verification of time constraints in embedded real-time systems should deal with three main issues: the resource restriction on the target hardware, the probe effect anomaly, and the lack of methods and support tools. Embedded systems may be resource constrained in processing power, energy consumption, memory space, board space, debug resources availability, production cost, and others. These restrictions impose limitations on what type of approaches can be used for testing. The probe effect anomaly occurs when the relative timing in the system is perturbed by observation [Thane 2000]. The lack of methods and support tools usually leads to customized test solutions, missing knowledge acquired in other designs. In an attempt to handle with the questions related to verification of time constraints in embedded real-time systems, the Laboratory of Embedded Systems Innovation and Technology (LIT), at the Federal Technological University of Paran (UTFPR), is working on the development of Dyretiva (Dynamic Real-Time Violation Analyzer) Method [Cadamuro 2005]. A typical Dyretiva test scenario is comprised of an embedded system running a preemptive kernel and a multi-threaded application (the target), an external hybrid monitor to collect trace data, and a set of tools to perform trace analysis. In this paper we present the Minimal Invasive Monitor (MIMO), which is the external hybrid monitor used in Dyretiva.

40

Technical Session 2 - Embedded Systems

2. Dyretiva Monitoring Approach

Dyretiva monitoring approach is based on the following needs of embedded real-time systems designs: portability, minimal intrusion, low cost, independence of the operating system and probe effect avoidance. To comply with the first three requirements, Dyretiva has adopted a hybrid monitoring approach with a standard and low complexity interface between target and monitor. Two alternate options to hybrid monitoring could be used: software-only monitoring and hardware-only monitoring. Software monitors are extensions of existing systems with the objective of registering relevant program events [Mahrenholz 2001]. They are portable and low cost, but highly intrusive. Hardware monitors observe the system under test using hardware probes without the need for changing the software. They act as measurement instruments, do not introducing any intrusion in the implementation under test, but are complex, not portable to different CPU architectures and costly. Operating system independence has been accomplished by the definition of standard physical and logical interfaces between target and monitor. Operating system instrumentation is performed through its sources and depends on a deep knowledge on kernel services and states. This task is assigned, on the typical case, to kernel designers. User application is instrumented using the Dyretiva source code instrumentation tool. Through the use of special format comments in the source code, the user can select which functions of the embedded application will be instrumented for a future time measurement. Later on, Dyretiva analysis tools will use the instrumentation description file, generated by the instrumentation tool, to identify user events in the trace. With the knowledge of the operating system and user application events, the analysis tools are able to identify the context of each monitored function call, calculating the time spend in each function correctly and independently of the occurrence of a preemption during the call. Probe effect avoidance in the Dyretiva monitoring approach can be accomplished by the careful instrumentation of the operating system and the selective instrumentation of the embedded application. Since it performs instrumentation only in the necessary locations, only a small part of the system time will be spend in executing instrumentation instructions, with the hybrid monitor performing time consuming monitoring tasks. Therefore, this approach makes it possible to keep in the released version the instrumentation used during system tests, making tested and released products the same from the temporal behavior point of view. A diagram of Dyretiva hybrid monitoring approach is presented in Figure 1. The target runs a preemptive real-time kernel and the embedded application. The kernel and the instrumented application send events to the monitor through target-monitor interface. The monitor is responsible for detecting and processing these events. Monitor operation is controlled by the user through a monitor access program running on the host computer. Host and MIMO communicate with each other via a monitor-host interface, implemented via a standard serial communications channel and a simple and reliable protocol. MIMO accepts commands from the monitor access program and processes events from the target.

IX Workshop on Real-Time Systems

41

Embedded Application

MIMO

Monitor Access Program

Target-Monitor Interface

Monitor-Host Interface

(Target)

(Hybrid Monitor)

(Host)

Figure 1. Dyretiva hybrid monitoring approach

2.1. Target-Monitor Interface The physical interface between target and monitor is defined by Dyretiva with 16 digital data lines (target output/monitor input), 1 data valid line (target output/monitor input) and a reference ground signal. With 16-bit data lines it is possible to signal up to 65,536 events, but in many cases this may become a limitation. In order to deal with this limitation, the available range has been divided in three logical groups: simple, data and operating system events. Simple events have values between 0x0000 and 0xD9FF, and are used by the user application to indicate events such as function start, function end, decision point and loop construction. User application instrumentation and the assignment of a value to an event are both performed by an automatic instrumentation tool. Data events have values between 0xE000 and 0xEFFF, and are used to identify both the variable being traced and its current value. A complete data event requires writing multiple 16-bit quantities to the monitor. Table 1 presents the meaning of the first 16-bit write of a data event in the target-monitor interface. sz field indicates the traced data size, with a '0' indicating a 16-bit variable and a '1' indicating a 32-bit variable. When a 16-bit variable is being traced one extra write containing the value of the variable should be performed. When a 32-bit variable is being traced two extra writes should be performed. The variable being traced is identified by the veid field, becoming possible to trace up to 2048 16-bit variables, and 2048 32-bit variables.
Table 1. Dyretiva data events format

Bit Numbers Value

15 - 12 0xE

11 sz

10 0 Veid

Operating system events have values between 0xF000 and 0xFFFF, and may use multiple writes to the interface as well. The format of the first write of an operating system event is presented in Table 2. size field indicates how many extra writes will be necessary to compose a complete event, and oseid contains the operating system event identifier. The meaning of an operating system event is assigned by operating system designers.

42

Technical Session 2 - Embedded Systems

Table 2. Dyretiva operating system events format

Bit Numbers Value

15 - 12 0xF

11 - 9 size

80 Oseid

All writes to the target-monitor interface comprised by more than one 16-bit quantity must be performed by the target atomically. Event numbers 0xDA00 to 0xDFFF are reserved. 2.2. Monitor-Host Interface The monitor-host physical interface is implemented via a standard serial communications channel, and a link layer provides data transfer reliability. The protocol running over link layer is called MIMO protocol. The host is the master of MIMO protocol and the monitor is the slave. The host can request three types of services to MIMO: configuration, command and data transfer. Configuration requests are used to set triggers and event sampling parameters. Command requests are used to start and stop acquisition in the target-monitor interface. Data transfer requests are used to upload an already collected trace data to the host.

3. MIMO Architecture
MIMO internal block diagram is depicted in Figure 2. All elements, except DDR SDRAM (Double Data Rate Synchronous Dynamic RAM) memory, are implemented in a Xilinx FPGA XC3S1500 [Xilinx 2006]. In this section we describe the most important aspects of MIMO.

Target

Sampler Dedicated Core Soft Processor

Serial commun.
DDR SDRAM

Host

DMA control Processor Bus

Figure 2. MIMO block diagram

In the upper left side of Figure 2 we have the Sampler Core MIMO hardware element. It is responsible for detecting and time stamping the events generated by the target, storing them locally in an internal dual ported RAM, entitled Input RAM. The contents of the Input RAM are periodically transferred to DDR SDRAM under DMA (Direct Memory Access) control. When DDR SDRAM is full, the Sampler Core stops data transfers to it. High level MIMO management tasks, like serial communications protocol and the control of internal data transfers, are performed by the Soft Processor.

IX Workshop on Real-Time Systems

43

MIMO has been built using Xilinx Embedded Development Kit (EDK), which provides libraries for many of the MIMO hardware components, such as the Soft Processor, the Serial Communication Controller, the DMA Controller and the DDR SDRAM Controller. A notable exception is the Sampler Core, which has been created to attend the specific needs of MIMO, and was developed in VHDL, a hardware description language [Pedroni 2004]. A block diagram of Sampler Core including its interfaces with other components is shown in Figure 3.

Sampler Core Target Input State Machine Input RAM

Bus Intrerface Unit (Slave)

Processor Bus

Figure 3. Sampler Core Block Diagram

The Sampler Core is a slave in the Soft Processor bus architecture. It can be controller by any of the bus masters, such as the Soft Processor or the DMA Controller. The Input RAM is a temporary storage for events collected from target and its timestamps. The Input State Machine is the hardware responsible for detecting, decoding, time stamping and storing events. A simplified diagram of the Input State Machine is presented in Figure 4.
rdy=1
stamp=H

Idle
head= sysop head= var

Sysop Check Var Check Single Check

Sysop

rdy=0

Sysop Stamp

stamp=L

stamp=H

rdy=1

Run
head= single

Var
rdy=0

Var Stamp stamp=L

stamp=H

End Event
stamp=L

Single

Single Stamp

Figure 4. Simplified diagram of the Input State Machine

Immediately after power on reset, Input State Machine moves from Idle to Run state, waiting for events from the target. When detecting a simple event, the machine reads the timestamp counter and moves from Run to Single Check state, which verifies space availability in the Input RAM to store the received event and its timestamp. With memory available, the machine moves from Single Check to the next two states, storing the event and the timestamp. Once the End Event state is traversed, the Input State Machine returns to the Run state to wait for more events.

44

Technical Session 2 - Embedded Systems

Another possibility for the Input State Machine is to receive a data event. When this occurs, the machine reads the timestamp counter and moves from Run to Var Check state, which verifies space availability in the Input RAM to store the whole data event and its timestamp. If the memory is available, the machine moves from Var Check to Var state to wait for, receive and store the data event. At the end of data reception, the machine moves from Var to Var Stamp state, storing the timestamp read at the Run state to the input RAM. Traversing the End Event state, the Input State Machine returns to the Run state again. The last path to be covered in the Input State Machine is the reception of an operating system event. Once an event of this type is detected, the machine reads the timestamp counter and moves from Run to Sysop Check state, which verifies the availability of Input RAM to store the whole operating system event and its timestamp. With memory available, the machine moves from Sysop Check to Sysop state to wait for, receive and store the operating system event. At the end of reception, the machine moves from Sysop to Sysop Stamp state and, after performing final tasks, returns to Run state. According to Dyretiva definitions, the timestamp counter should be incremented in steps of 20ns, and a timestamp value should be related to the last received event. Following this guideline, the timestamp counter is reset when its value is read at Run state. The non-time-critical MIMO tasks were implemented in software, using the Soft Processor implemented inside FPGA. The MIMO embedded software was developed in ANSI C programming language, and uses a kernel provided by Xilinx with the Soft Processor hardware. The kernel provides the basic support for concurrent programming, as well as inter-thread communication and synchronization mechanisms. The main parts of the MIMO embedded software are: device drivers for hardware components, an implementation of the MIMO protocol, and the coordination of hardware related MIMO tasks.

4. Performance Considerations
Although the input RAM is just 2,048 bytes long, the Sampler Core should continue detecting and storing events during data transfers between input RAM and DDR. To solve this problem the input RAM has been divided in two 1,024 bytes halves. While events are being collected by the Sampler Core to one half of input RAM, the other half can be transferred to DDR. The worst case for Sampler Core in terms of memory bandwidth usage is the continuous arrival of simple events. To detect, timestamp and store these simple events, the Sampler Core requires six clock cycles and uses six bytes of the input RAM, two for the event itself and four for the timestamp. The current clock rate provided to this sampler machine is 75MHz, leading to 80ns to complete the detection and caching of a simple event. Under continuous arrival, a half of input RAM will be filled in 13.65s, and we can deal with a maximum of 12,500,000 events per second. The transfers between Input RAM and DDR are performed by the DMA controller, using DDR burst mode. A logic analyzer tool for inspecting FPGA internal signals has shown that in our system a 64 byte burst transfer is accomplished in

IX Workshop on Real-Time Systems

45

986.7ns, including the data transfer itself and the interval between two consecutive bursts. In this case the transfer from half of the input RAM to DDR takes 15.79s. This value is about 2s greater than the time needed to fill the input RAM in the worst case, and can be considered the limitation for data transfers. With 30MB of DDR available for storing events, and the generation of simple events being the only target activity, the whole DDR event buffer will be filled in a half second. This value is an absolute maximum rate and does not occur in real situations. To check these values in practice we have performed experiments with a real target connected to MIMO: an evaluation board containing a Freescale MPC8xx Power processor [Freescale 2007] running at 48MHz. The application under test was an implementation of the Blowfish cryptography algorithm [Schneier 1994] running continuously in an infinite loop. The start and the end of encrypt and decrypt functions were instrumented, and these functions were called in a loop to encrypt/decrypt every 8 bytes of data. In this scenario, we have detected the generation of two simple events (start and end of function) to MIMO at every 30s, using 12 bytes of the input RAM. This situation fills all the 30MB of DDR available for storing events in 78.6 seconds. Improvements to the MIMO performance can be accomplished by reducing the number of clock cycles used in each event acquisition performed by the Sampler Core state machine, and by increasing the clock rate of both Sampler Core and DDR SDRAM.

5. Conclusions and Future Work

In this paper we have presented MIMO, a hybrid monitor designed to be used in the verification of time constraints of embedded real-time systems. It follows Dyretiva Method design principles: portability, minimal intrusion, low cost, independence of the operating system and provision for probe effect avoidance. We have implemented a MIMO proof of concept with programmable logic in a low cost FPGA, and it was necessary to develop a dedicated and complex hardware component to fulfill event sampling requirements. We calculate the MIMO maximum supported rates. Although we reach the conclusion that in the worst case a 30MB buffer will be filled in only half a second, a practical experiment has shown that two minutes of tracing in a real embedded system is feasible. A feature provisioned by MIMO protocol that could be implemented in a future release of Sampler Core is the trigger facility. A trigger is an event set by user to start acquisition autonomously. A sophisticated trigger facility may include an indication of estimated event position in the buffer. A current MIMO limitation is the operating model: acquiring first and transfer data to host later on. An interesting improvement to MIMO could be the implementation of a high speed serial communications channel, like a Gigabit Ethernet. Such a new version of MIMO could transfer events to the host during acquisition, improving the trace length limits without requiring more local storage. This approach may also require more processing power from the Soft Processor.

46

Technical Session 2 - Embedded Systems

References
Burns, A. and Wellings, A. (1997) Real-Time Systems and Programming Languages, Addison Wesley Longman, 2nd edition. Thane, H. and Hansson, H. (2000) Using Deterministic Replay for Debugging Distributed Real-Time Systems, In 12th Euromicro Conference on Real-Time Systems. Mahrenholz, D. (2001) Minimal Invasive Monitoring, In Proceedings of the 4th International Symposium on Object-Oriented Real-Time Distributed Computing (ISORC01), pp. 251-258, IEEE Computer Society. Cadamuro Jr., J. (2005) Dyretiva: Um Mtodo para Teste Dinmico de Sistemas em Tempo Real, DSc Thesis Proposal Report, CPGEI / UTFPR, Curitiba, Brazil. Xilinx, Inc. (2006) Spartan-3 FPGA Family: Complete Data Sheet, Product Specification v2.1, Document DS099-4, April. Pedroni, V. A. (2004) Circuit Design with VHDL, The MIT Press. Freescale Semiconductor, Inc. (2007) PowerQUICC Communication Processors, http://www.freescale.com/webapp/sps/site/homepage.jsp?nodeId=0162468rH3bTdG Jk19, March. Schneier, B. (1994) Description of a New Variable-Length Key, 64-bit Block Cipher (Blowfish), In Cambridge Security Workshop Proceeding, pp. 191-204, SpringerVerlag.