Kernel Structure and

Infrastructure
David Ferry, Chris Gill, Brian Kocoloski, Marion Sudvarg
CSE 422S - Operating Systems Organization
Washington University in St. Louis
St. Louis, MO 63130

1
 Kernel vs. Application Coding
Two major differences:
– The core kernel has no standard libraries
– The core kernel must be a monolithic, statically linked
library

No standard libraries:
– No libc (malloc, pthreads, string handling, etc.)
– Partly because of chicken/egg situation
– Also, standard libraries can be too slow

No dynamic loading of shared libraries for the core

kernel
CSE 422S – Operating Systems Organization 2
 Two Big Problems
1. Programmers rely on “library” functions for
efficiency and correctness
– E.g., data structures, calls to sleep, etc.
– What libraries to use? (hint: not glibc)
• Kernel code has to be entirely self-contained
• Also, chicken+egg problem, as glibc functions may in turn
invoke the kernel via system calls

2. A totally static kernel would be enormous

– About 20 million lines of code in 2015
– Most of this is hardware drivers that are never used
on any given platform

CSE 422S – Operating Systems Organization 3

 First Solution: Kernel “Libraries”
Kernel libraries re-implement a lot of the functionality
programmers expect in user space
– Are statically compiled into the kernel
– Automatically available just by including relevant header
– Built to be kernel-safe (sleeping, waiting, locking, etc. is
done properly)

Features:
– Utilities: kmalloc, kthreads, string parsing, etc.
– Containers: hash tables, binary trees etc.
– Algorithms: sorting, compression

Mostly found under /lib and /include/linux

CSE 422S – Operating Systems Organization 4

 Kernel “Libraries”

Many kernel “libraries” have clear analogues

to user space libraries:
- malloc/free vs kmalloc/kfree
- pthread_create vs kthread_create
- sleep/usleep/nanosleep (user) vs
msleep (kernel)

Some, however, are a bit different:

- e.g., linked list implementation

CSE 422S – Operating Systems Organization 5

 Example: Linked Lists
(LKD pg. 88)

struct fox {
unsigned long tail_length;
unsigned long weight;
bool is_fantastic;
};

If you wanted a list of structs of type struct_fox:

struct fox { struct fox {

unsigned long tail_length; unsigned long tail_length;
unsigned long weight; unsigned long weight;
bool is_fantastic; bool is_fantastic;
struct fox * prev; struct list_head list_node;
struct fox * next; };
};

CSE 422S – Operating Systems Organization 6

 Case 1: Manually Linked Together
struct fox {
unsigned long tail_length;
unsigned long weight;
bool is_fantastic;
struct fox * prev;
struct fox * next;
};

struct fox { struct fox { struct fox {

unsigned long tail_length; unsigned long tail_length; unsigned long tail_length;
unsigned long weight; unsigned long weight; unsigned long weight;
bool is_fantastic; bool is_fantastic; bool is_fantastic;
struct fox * prev; struct fox * prev; struct fox * prev;
struct fox * next; struct fox * next; struct fox * next;
}; }; };

Disadvantage: need to “roll your own code” for each list you create
 Duplicate code throughout the kernel
 Introduce bugs
 Lose optimizations (placement within cache lines, etc.)

CSE 422S – Operating Systems Organization 7

Case 2: Embedding Fox in list_head
A list-able struct must contain struct list_head

struct list_head {
struct list_head *next;
struct list_head *prev;
};

If you wanted a list of structs of type data:

struct fox{
unsigned long tail_length;
unsigned long weight;
bool is_fantastic;
struct list_head list_node;
};

CSE 422S – Operating Systems Organization 8

 Source: https://notes.shichao.io/lkd/ch3/

CSE 422S – Operating Systems Organization 9

 Example: Linked Lists
Initializing a list dynamically:

struct fox *new_fox;

new_fox = kmalloc(sizeof(struct fox), GFP_KERNEL);
new_fox->tail_length = 40;
new_fox->weight = 6;
new_fox->is_fantastic = false; List head is initialized empty,
i.e. points to itself
INIT_LIST_HEAD(&(new_fox->list_node));

Or statically at compile time:

static LIST_HEAD(fox_list_head);

CSE 422S – Operating Systems Organization 10

 Example: Linked Lists
Q: list_node.next, list_node.prev point to other
list_head. So, how do I get access to the struct fox *
embedded within?
struct fox {
unsigned long tail_length;
unsigned long weight;
bool is_fantastic;
struct list_head list_node;
};

list_node.prev list_node.next

struct list_head { struct list_head {

struct list_head * prev; struct list_head * prev;
struct list_head * next; struct list_head * next;
}; };

CSE 422S – Operating Systems Organization 11

 Example: Linked Lists
Casting out from list_node to containing structure:

struct fox * fox;

struct list_head * node;

fox = list_entry(node, struct fox, list_node);

/*
list_entry(
arg 1: Pointer to list_head structure,
arg 2: Type of outer “containing structure”,
arg 3: Struct member of list_head within containing
structure
)
*/

CSE 422S – Operating Systems Organization 12

 Example: Linked Lists
Functions may take pointers to list nodes or pointers to
the list head:

Adding:

struct data *new_data;

list_add(&new_data->list, &data_list_head);

Deleting:

list_del(&new_data->list);
kfree(new_data); //if dynamic

CSE 422S – Operating Systems Organization 13

 Example: Linked Lists
Iterating:

struct list_head ptr;

list_for_each(ptr, data_list_head){
//ptr points to each list structure
}

struct data *data_ptr;

list_for_each_entry(d_ptr, data_list_head, list){
//d_ptr points to each data structure
}

Also:

list_for_each_entry_reverse()
list_for_each_entry_safe() //for modifying list elements

CSE 422S – Operating Systems Organization 14

 Example: Linked Lists
See also:
– /include/linux/list.h
– /include/linux/types.h

Moral of the story:

Always search for functionality before
writing it yourself.

CSE 422S – Operating Systems Organization 15

 Second Solution:
Loadable Kernel Modules
Kernel modules are kernel code that can be
loaded dynamically:
– Can be loaded/unloaded whenever
– Runs in kernel mode
– Can access exported kernel variables and
functions
– Can export variables and functions to kernel or
other modules

CSE 422S – Operating Systems Organization 16

 Why Kernel Modules
Linux is a monolithic kernel. All functionality is
compiled into the same static binary that is
loaded into memory on boot.

Without modules, the entire kernel would

need to be loaded into memory to boot a node.
Problems?
 Waste of memory (embedded systems)
 Slower boot time
 Larger trusted computing base (TCB), more room
for bugs

CSE 422S – Operating Systems Organization 17

What are Kernel Modules used for?
• What pieces of code do we think might not
be needed on every system that the kernel
boots on?
1. Device Drivers
2. Architecture-specific code

• Lines of code (just .c files) for:

1. Device Drivers: 1,583,159
2. Everything else: 1,003,777
• (Includes all of the architectures that we’re not using!)

CSE 422S – Operating Systems Organization 18

What are Kernel Modules used for?
• Beyond space savings, what else might
modules be useful for?
1. “Out-of-tree” functionality that is not
accepted into “mainline” kernel
2. Allowing users to load custom functionality at
runtime
3. Configuring/patching a running system
without requiring a reboot
4. Your lab assignments in this class 

CSE 422S – Operating Systems Organization 19

 Module Implementation
Must define:
– An initialization function called on load
– An exit function called on unload

The init function must be self contained!

– Must unwind actions if initialization cannot complete
successfully
– E.g. if you kmalloc() space but don’t free it, that
physical memory is now lost (until system restart)

You can also pass parameters to modules at load time.

CSE 422S – Operating Systems Organization 20

 Loading and Unloading Modules
Today we will use insmod and rmmod
– Must use sudo to invoke these on your Rpi
ERROR: could not insert module jiffies_module.ko:
Operation not permitted
– Cannot unload a module that’s not loaded
ERROR: Module jiffies_module is not currently
loaded
– Cannot load a module that’s already there
ERROR: could not insert module simple_module.ko:
File exists

CSE 422S – Operating Systems Organization 21