EE514 Real-Time Computing

Real-Time Linux
Department of Electrical Engineering
The Hong Kong Polytechnic University

EE514 Real-Time Computing

Real Time Linux Operating Systems

Swaminathan Sivasubramanian
Dependable Computing & Networking Laboratory
Iowa State University
swamis@iastate.edu

EE514 Real-Time Computing

Linux as a Real-Time OS
Traditional

RT Systems used custom built

systems which were not extensible I.e.
tough to develop new applications
However, as technology improved, generic
real-time OS became acceptable
In OS suited for extensible development
Linux looks more appealing

EE514 Real-Time Computing

Why Linux?

Linux (and its Real Time versions) are

free!!
Linux (and its Real Time versions) are
Open Source!!
Easy for developing RT applications

EE514 Real-Time Computing

Why not Linux?

Linux

didnt have any corporate support

until now
Linux, is a very good general purpose
operating system, but not so for real-time
OS (Why??)
Because, the design motive of a
conventional OS and RTOS is different

EE514 Real-Time Computing

 Linux as your real-time

solution?
Could increase priority for real-time tasks
and assume they get scheduled
Problem Linux optimizes average case
whereas an RTOS should work under worst
case assumptions

EE514 Real-Time Computing

Linux A Simplified View

EE514 Real-Time Computing

Linux conflicts with RT constraints

Coarse grained synchronization long intervals when a

task has exclusive use of data ( as fine grained leads to
lot of overhead reducing the average case performance)
Linux batch all operations for efficient use of H/W

E.g freeing a list of pages when memory is full

reducing the worst case performance

Linux doesn't preempt low-priority task during system call

Linux will make high priority tasks wait for low priority to
release resources

EE514 Real-Time Computing

Real Time Linux approaches

Modify

the current Linux kernel to

guarantee RT constraints

Used by KURT

Make

the standard Linux kernel run as a

task of the real-time kernel

Used by RT-Linux, RTAI

EE514 Real-Time Computing

Modifying Linux kernel

Advantages

Most problems, such as interrupt handling,

already solved
Less initial labor

Disadvantages

No guaranteed performance
RT tasks dont always have precedence over
non-RT tasks.

EE514 Real-Time Computing

10

Running Linux as a process of a second

RT kernel
Advantages

Can make hard real time guarantees

Easy to implement a new scheduler

Disadvantages

Initial port difficult, must know a tremendous

amount about underlying hardware
Running a small real-time executive is not a
substitute for a full fledged RTOS

EE514 Real-Time Computing

11

RTAI Overview
Based

on Real-Time Hardware Abstraction

Layer (HAL) (also used in Windows NT)
HAL exports Linux data and functions
related to hardware
HAL defines an interface between RTAI
and Linux

The Windows NT hardware abstraction layer (HAL) refers to a layer of software that
deals directly with your computer hardware.

EE514 Real-Time Computing

12

RTAI A Simplified View

EE514 Real-Time Computing

13

RTAI Overview (continued)

Software

architecture

Interface to Linux hardware management

(HAL)
3 basic components: dispatcher, scheduler,
fifos
1 interface used in user tasks to initialize and
start the components

RTAI

is basically an interrupt dispatcher

(reroutes interrupts to Linux if necessary)
e.g: Disk interrupt

EE514 Real-Time Computing

14

RT-Linux Overview
Open

source Linux project

Supports x86, PowerPC, Alpha
Patch of the regular Linux kernel (simply
install the patch and recompile the kernel)
Provides an RT API for developers
Runs Linux kernel as lowest priority
process

EE514 Real-Time Computing

15

RT-Linux Task Structure

EE514 Real-Time Computing

16

RT-Linux Interrupt Dispatcher

EE514 Real-Time Computing

17

RT-Linux Overview (continued)

RT

tasks are coded as modules

Modules are inserted and removed at
users discretion
Extremely good at handling periodic tasks
Communicates with non-RT kernel and
other RT tasks via fifo queues
Tools are provided for graphical analysis of
RT execution
EE514 Real-Time Computing

18

Problems with RT-Linux

Currently

no support for aperiodic tasks

Not very useful for complex RT systems
Currently limited to simple problems

EE514 Real-Time Computing

19

Montavista A Faster Response Linux

Kernel
Montavista

Inc. provides a linux solution for

embedded systems
The solutions aim is to make the Linux
kernel fully preemptable
It identifies the points where priority
inversion occurs in Linux and makes those
points fully preemptable
A good embedded solution not a complete
RT solution.
EE514 Real-Time Computing

20

Real-Time Linux

A class of Linux systems which could meet the real-time

requirements.

There are a number of such real-time Linux systems

available:

RTLinux hard real-time

RTAI hard real-time

MontaVista soft real-time

and many more

Many of them are developed as research projects and

were released to public freely.

Some of them are commercialized though.

EE514 Real-Time Computing

21

RTLinux or RTCore

The first real-time Linux system developed as a research

project over a decade ago.

Subsequently commercialized to become a real-time core

(RTCore) of a commercial Linux called Wind River Linux.

RTLinux or RTCore is a hard real-time microkernel that

runs the entire Linux operating system as a fully
preemptable process.

The technologies developed to enable the real-time

operation are patented and cant be used freely.

EE514 Real-Time Computing

22

RTAI

RTAI stands for Real-Time Application Interface and is a

Linux extension to hard real-time application.

It is initially developed as a research project in Italy for

enhancing the RTLinux functionalities at that time.

Subsequently, new technologies were invented to replace

all the RTLinux code adopted from the initial RTLinux
prototype to overcome any patent problems.

The new technologies is collectively referred as ADEOS.

Current version is 3.8.1 released in May 2010.

Still actively under development and maintained by the

original developers in Italy and the Internal community.

EE514 Real-Time Computing

23

Beginning of Linux as a UNIX clone

EE514 Real-Time Computing

24

Linux System

Linux kernel

In Sept 1991, Linus Torvalds, a second year student of Computer

Science at the University of Helsinki, developed the preliminary
kernel of Linux, known as Linux version 0.0.1, and released to the
Internet.

Version 1.0, 2.0, 2.2, 2.4, and 2.6 were released in 1994, 1996,
1999, 2001, and 2003, respectively.

Current version is 2.6.37.2 released on 24 Feb 2011.

Linux system

Software from the GNU project + Linux kernel = Linux system

Redhat, Fedora, CentOS, Ubuntu, etc. are popular variants of

Linux distributions.

EE514 Real-Time Computing

25

Evolution of Linux

Early Linux is Not Designed for Real-Time Processing

Early Linux (1.x Kernel) installations on retired Windows PCs

Old/Obsolete hardware useful under Linux due to efficiency of O/S

Linux outperformed Windows in reliability and uptime (still does)

Linux Design: Fairness, Throughput and Resource-Sharing

Basic Unix development design principles applied in Kernel

Heavily (over)-loaded systems continue to make progress

Does not drop network connections or starve users / applications

Fairness- and Resource-Sharing Design is Linux's Strength

contributed to make Linux competitive and popular in the enterpriseserver and development-application environments

Gave rise to RedHat and others.

Essential to the evolution of Linux, endemic of UNIX legacy

EE514 Real-Time Computing

26

Why Linux in Real-Time Systems?

Not because of the Kernels Real-Time Performance!

UNIX-legacy Operating Systems were designed with operating

principles focused on throughput and progress

User tasks should not stall under heavy load

System resources must be shared fairly between users

Fairness, progress and resource-sharing conflict with the

requirements of time-critical applications

VIP vs. General Admission

UNIX systems (and Linux) are historically not Real-Time OS

Linux has lagged many commercial Unix's in Real-Time

performance-enhancement and Real-Time capabilities

Solaris, LynxOS, QNX

SCO

EE514 Real-Time Computing

27

Real time computing requirements

Scheduling
Hardware Interface
Communication and
Synchronization
Memory Management
Task Management

Timing
Configuration
External
Communications
Error Reporting
Embedded Features
File system
Reliability

28
EE514 Real-Time Computing

Why Real-Time in Linux Systems (Embedded)

Most Important Factors Inhibiting Linux Adoption

Data from VDC, Linuxs Future in the Embedded Systems Market, June 2004
EE514 Real-Time Computing

29

Real-Time in Handheld & Embedded Systems

Cost / Performance / Power / Weight Compromise

Competitive, High-Volume, Low-margin Markets

Maximum Feature-set, Add-ons, Responsive UI feel

Device specs: minimal CPU & Memory & Battery Powered

Minimal CPU = High CPU utilization

High CPU load + Time-Critical functionality RT specs

Real-time Requirements will never be alleviated by

Improvements in Hardware Performance / Efficiency

Software utilizing latest hardware technologies easily keep up with,

and usually out-paces, advances in hardware technology

If you don't believe that, go shopping (for a mobile phone)

EE514 Real-Time Computing

30

Real-Time Linux 2.6 Enablers

Pro-Audio Performance Requirements

Audio Community Involved in Kernel-Preemption since 2.2

Audio Community strongly Endorsing RT technology

Embedded Application Domain

Single-Chip, Mobile Applications (Wireless / Cellular Handsets)

Predictable OS performance eliminates HW design uncertainty

Multimedia Carrier (QOS) Application Domain

Reliable Prototyping and Improved Product Scheduling

Telephony, Audio / Video / Multimedia / Home Entertainment

Fine-Granular Preemption improves SMP scalability

Mainstreaming of SMP Technology

Dual / Quad / Octa - Core Intel, AMD, PPC, Arm

EE514 Real-Time Computing

31

Real-Time and Linux Kernel Evolution

Gradual Kernel Optimizations over Time

SMP Critical sections (Linux 2.x)

Low-Latency Patches (Linux 2.2)

Preemption Points / Kernel Tuning (Linux 2.2 / 2.4)

Preemptible Kernel Patches (Linux 2.4)

Fixed-time O(1) Scheduler (Linux 2.6)

Voluntary Preemption (Linux 2.6)

In 2003-04 Linux 2.6 RT Technology Regressed

Early Linux 2.6 Real-Time Performance was worse than 2.4

Kernel Performance

Audio Community and others balked at moving to 2.6 Kernel

Base

EE514 Real-Time Computing

32

Real-Time Inhibitor: Critical Section Locking

Linux 2.6 Kernel Critical Sections are Non-Preemptible

Critical sections protect shared resources, e.g. hardware

registers, I/O ports, and data in RAM

Critical sections are shared by Processes, Interrupts and CPUs.

Effective protection is provided by the Spin-Lock Subsystem

Critical sections must be locked and unlocked

Locked critical sections are not preemptible

Linux 2.6 Kernel has 11,000 critical sections

Exhaustive Kernel testing to identify worst-case code paths

Labour-intensive cleanup of critical sections

No control over 3rd party drivers

Worst-case after cleanup still not acceptable

Maintenance, community education, policing / regression testing

EE514 Real-Time Computing

33

Real-Time Inhibitor: Interrupt Handlers

Linux 2.6 Kernel: Unbounded IRQ subsystem latencies

Task-Preemption latency increases with hardware-interrupt load

Interrupts cannot be preempted

No Priorities for Interrupts

IRQ Subsystem always preempts tasks unconditionally

Unbounded SoftIRQ subsystem (Bottom Half Processing)

Activated by HW IRQs (Timers, SCSI, Network)

SoftIRQs re-activate, iterate

Driver-level adaptations

Network Driver NAPI adaption reduces denial of service (D.o.S.)

effects of high packet loads

EE514 Real-Time Computing

34

Real-Time Inhibitor: Legacy Locking

Existing Locking Subsystems are not Priority-Aware

System semaphore

Counting semaphore used to wake multiple waiting tasks

No support for priority inheritance

No priority ordering of waiters

Big Kernel Lock (BKL)

Originally non-preemptible, now preemptible using system semaphore

Can be released by blocking tasks, re-acquired upon wake-up

No priority-awareness, or priority inheritance for contending tasks

RCU (Read-Copy-Update) Locks in Network subsystem

Read-optimized cached locking requiring race-free invalidation

Read Write Locks

Classical blocking / starvation issues with no priority awareness

EE514 Real-Time Computing

35

The Fully Preemptible Linux Kernel

Dramatic Reduction in 2.6 Preemption Latencies

Multiple Concurrent Tasks in Independent Critical Sections

Generally Fully Preemptible No Delays

Design Flexibility

Provides Full Access to Kernel Resources to RT Tasks

Supports existing driver and application code
User-space Real-Time

Optimization Flexibility

Non-preemptible: Interrupt off paths and lowest-level interrupt management

Non-preemptible: Scheduling and context switching code

RT Tasks designed to use Kernel-resources in managed ways can

reduce or eliminate Priority-Inheritance delays

Adequate Instrumentation

Latency timing, latency triggers & stack tracing, histograms

EE514 Real-Time Computing

36

Kernel Evolution: Preemptible Code

Kernel 2.0

Kernels
2.2-2.4

Preemptible
Kernel 2.4
Kernel 2.6
Real-Time
Kernel 2.6
Preemptible
EE514 Real-Time Computing

Non-Preemptible
37

Real-Time Linux 2.6 Performance

Real-Time Linux 2.6 Kernel Performance

Far exceeds most stringent Audio performance requirements

Enables sub-millisecond control-loop response

Enables Hard Real Time for qualified RT-aware Applications

SMP Kernel Performance

SMP-safe code is by definition preemptible

Any code that allows concurrent execution by multiple CPUs, also

allows context switching and therefore preemption

Increased preemptible code surface in the Kernel also increases

SMP throughput / efficiency

EE514 Real-Time Computing

38

Real-Time Linux 2.6 Performance

(Cont)
No Preemption

Preemptible

MontaVista Linux 4.0

Target machine:

CPU utilization during test:

Intel Celeron 800 MHz

Real-Time
Preemption

100% most of the time

Test Duration:

20 hours

EE514 Real-Time Computing

39

RTLinux / RTAI Architecture

Users task

System libraries

Real Time task

Linux kernel

Drivers
I/O
Real-time Scheduler

Software interrupts

Real-time plugin

I/O

Hardware interrupts
HARDWARE

Hardware abstraction layer

real-time interrupt dispatcher, real-time scheduler

Inter processes communication services

EE514 Real-Time Computing

40

Interrupt Latency

The time stamp is frozen when the interrupt is generated

The current time is read when the interrupt is

acknowledged

The difference gives the interrupt latency

Dual-Xeon
3.2GHz

EE514 Real-Time Computing

Dual-core
Opteron
2GHz

41

Freescale dual-kernel, real-time solutions

(BSP) for Linux (Case studies)

Solutions provide sub-20 microsecond precision with

hard, real-time scheduling guarantees but at the
expense of increased programming effort and
memory protection
These techniques offer less than 250 sec latency
performance for greater than 99.5 percent of all
scheduled task under demanding system test loads
Under real-world conditions, performance is near the
99.5 percent, 100 sec mark

BSP : board support service

EE514 Real-Time Computing

42

Linux kernel improvements for real time

Features and performance enhancement for

upgrading standard Linux kernel to challenge real
time products and applications:

Enhanced schedulers
Virtual memory
Shared memory
Portable operating system interface X (POSIX) timers
Real-time signals
POSIX asynchronous I/O
POSIX threads
Quality of service capabilities
Low latency/preemptable kernel modifications

EE514 Real-Time Computing

43

Quality of sercice

As for quality of service, Linux v2.4 offers a

sophisticated component for bandwidth
management called Traffic Control. The
component supports methods for classifying,
prioritizing and limiting both incoming and
outgoing traffic. Linux can perform:

Limit bandwidth for certain computers

Help fairly share bandwidth
Protect the Internet from abuses
Restrict access and do routing based on user ID, MAC
address, source IP address and so on

EE514 Real-Time Computing

44

Linux network subsystem (Traffic

control)

EE514 Real-Time Computing

45

Queueing discipline- traffic control

Queueing

discipline provides a method to

enqueue a packet

EE514 Real-Time Computing

46

Low-Latency patches for Linux

The kernel latency is a quantity used to measure
the difference between the theoretical schedule
and the actual one
In a standard Linux kernel, the maximum latency
is equal to the maximum length of a system call
plus the processing time of all the interrupts that
fire before returning to user mode. This value can
be as large as 100 ms

EE514 Real-Time Computing

47

Low-Latency Linux (Red-Linux, some

Real-time
Linux)

This approach corrects the monolithic structure by

inserting explicit rescheduling points inside the
kernel. In this approach, when a thread is executing
inside the kernel it can explicitly decide to yield the
CPU to some other thread
In this way, the size of non-preemptable sections is
reduced, thus decreasing the latency
The consistency of kernel data is enforced by suing
cooperative scheduling
Since the low-latency patch has been carefully handtuned for quite a long time, it performs surprisingly
well

EE514 Real-Time Computing

48

Preemptable Linux (used in most realtime

systems)

Removes the constraint of a single execution flow

inside the kernel. It is not necessary to disable
preemption when an execution flow enters the kernel
To support full kernel preemptability, kernel data must
be explicitly protected using mutexes or spinlocks
The Linux preemptable kernel pathc, maintained by
Robert Love and sponsored by MontaVista

EE514 Real-Time Computing

49

Evaluation

EE514 Real-Time Computing

50

Probability density function of the

latency in the low-latency kernel

EE514 Real-Time Computing

51

Summary of the Freescale case studies

Linux is rapidly evolving to include more and more

features that have explicit value to the embedded
developer
Key enhancements to the Linux kernel, which
perform identically for all practical purposes
The developers choice regarding which should be
implemented can be based on independent criteria
including the ability to update/modify existing device
drivers for compatibility with the real-time
implementation, and the availability of each real-time
technique for the selected architecture and kernel
version

EE514 Real-Time Computing

52

References
RT-Linux

: http://www.rtlinux.org

RTAI

:
http://www.aero.polimi.it/projects/rtai/contri
b.htm
Montavista: http://www.mvista.com
Linux as a real-time operating system
Freescale semiconductor, David Beal,
Nov/2005

EE514 Real-Time Computing

53

EE514 Real-Time Computing

Real Time Linux

54

EE514 Real-Time Computing

Real Time Linux

55